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Overview 


Community colleges offer a pathway to the middle class for low-income individuals. Although 
access to college has expanded, graduation rates at community colleges remain low, especially for 
students who need developmental (remedial) courses to build their math, reading, or writing skills. 
Many reforms have been found to help students in the short term, but few have substantially boosted 
college completion. The City University of New York’s (CUNY’s) Accelerated Study in Associate 
Programs (ASAP), launched in 2007 with funding from the New York City Center for Economic 
Opportunity, is an uncommonly comprehensive and long-term program designed to help more 
students graduate and help them graduate more quickly. 

ASAP represents both an opportunity and an obligation for students. It was designed to address 
multiple potential barriers to student success and to address them for up to three years. ASAP 
requires students to attend college hill time and encourages them to take developmental courses 
early and to graduate within three years. The program provides comprehensive advisement from an 
adviser with a small caseload and enhanced career services and tutoring. ASAP offers blocked or 
linked courses for the first year and offers a seminar for the first few semesters, covering topics such 
as goal-setting and study skills. The program provides a tuition waiver that fills any gap between 
financial aid and college tuition and fees. It also provides free MetroCards for use on public trans- 
portation, contingent on participation in key program services, and free use of textbooks. 

This report presents results from a random assignment study of ASAP at three CUNY community 
colleges: Borough of Manhattan, Kingsborough, and LaGuardia. Low-income students who needed 
one or two developmental courses were randomly assigned either to a program group, who could 
participate in ASAP, or to a control group, who could receive the usual college services. Comparing 
the two groups’ outcomes provides an estimate of ASAP’s effects. Key findings from the report 
include the following: 

• ASAP was well implemented. The program provided students with a wide array of services 
over a three-year period, and effectively communicated requirements and other messages. 

• ASAP substantially improved students’ academic outcomes over three years, almost 
doubling graduation rates. ASAP increased enrollment in college and had especially large 
effects during the winter and summer intersessions. On average, program group students earned 
48 credits in three years, 9 credits more than did control group students. By the end of the study 
period, 40 percent of the program group had received a degree, compared with 22 percent of the 
control group. At that point, 25 percent of the program group was enrolled in a four-year school, 
compared with 17 percent of the control group. 

• At the three-year point, the cost per degree was lower in ASAP than in the control condi- 
tion. Because the program generated so many more graduates than the usual college services, 
the cost per degree was lower despite the substantial investment required to operate the program. 

ASAP’s effects are the largest MDRC has found in any of its evaluations of community college 
refomis. The model offers a highly promising strategy to markedly accelerate credit accumulation 
and increase graduation rates among educationally and economically disadvantaged populations. 
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Preface 


Community colleges provide relatively affordable postsecondary instruction to millions of 
students across the country, and their critical role in helping build the nation’s workforce has 
gained increasing recognition. Unfortunately, many community college students never earn a 
degree. Completion rates are especially low for students who enter college without all the math, 
reading, or writing skills they need to do college-level work. Numerous refomis have been tried 
to help students with developmental education needs succeed, but few have substantially 
boosted college completion. 

In 2007, the City University of New York (CUNY), with the support of the New York 
City Center for Economic Opportunity, launched Accelerated Study in Associate Programs 
(ASAP) to encourage and support community college students to attend school full time and 
graduate. The exceptionally ambitious program provides a rich array of financial assistance, 
special courses, enhanced advising, and other support services for three full years. 

This report presents important findings from MDRC’s random assignment evaluation 
of ASAP at three CUNY community colleges. The evaluation targeted low-income students 
who needed one or two developmental courses to build their math, reading, or writing skills. 
ASAP’s effects are the largest MDRC has found in any of its evaluations of community college 
refonns. After three years, the program substantially increased full-time enrollment, accelerated 
credit accumulation, and almost doubled the rate of graduating with an associate’s degree. It 
also increased the likelihood that students would transfer to a four-year school. Positive effects 
were found for all of the subgroups of students examined in the evaluation. The evaluation also 
found that, even though ASAP required a substantial investment, the cost per degree was lower 
among students in the program than it was for those receiving the usual college services. 

ASAP’s effects after three years signal the great promise of comprehensive, extended 
interventions to substantially improve outcomes for community college students. ASAP’s 
effects are especially notable given that they were for a group of students who entered college 
with developmental education needs. ASAP shows that such students can succeed with the right 
combination of services and supports, and without changing what happens inside the classroom. 

In light of ASAP’s success, CUNY is continuing to expand the program, with a goal of 
serving over 13,000 students by fall 2016. CUNY and MDRC are working together to develop 
evaluations of ASAP replications at interested community colleges. ASAP is receiving attention 
across the nation, and rightly so: It is a model program all colleges should consider. 

Gordon L. Berlin 
President, MDRC 
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Executive Summary 


Community colleges offer a pathway to the middle class for low-income individuals. Although 
access to college has expanded in recent decades, graduation rates at community colleges 
remain low, especially for students who need developmental (remedial) courses to build their 
basic skills. Nationwide, only about 15 percent of students with developmental education needs 
attending a two-year college earn a degree or certificate within three years. 1 Many refonns have 
been found to help students in the short tenn, but few have meaningfully boosted college 
completion. This report discusses a program that made a big difference for students and sub- 
stantially increased graduation rates. Accelerated Study in Associate Programs (ASAP), 
operated by the City University of New York (CUNY), one of the nation’s largest public urban 
university systems, is an uncommonly comprehensive and long-tenn program designed to help 
more students graduate and help them graduate more quickly. Launched in 2007 with funding 
from the New York City Center for Economic Opportunity, ASAP has been implemented at six 
of CUNY’s seven community colleges. 

ASAP presents students with both an opportunity and an obligation. Designed to ad- 
dress multiple potential barriers to student success, ASAP provides structure and support for up 
to three years through the following key components: 

• Requirements and messages: Students are required to attend college full 
time and are encouraged to take developmental courses early and to graduate 
within three years. 

• Student services: Students receive comprehensive advisement from an 
ASAP-dedicated adviser with a small caseload, career infomiation from an 
ASAP-dedicated career and employment services staff member, and ASAP- 
dedicated tutoring services. 

• Course enrollment: Students may enroll in blocked or linked courses (two 
or more courses grouped together with seats reserved for ASAP students) in 
their first year. Students also enroll in an ASAP seminar during their first few 
semesters covering topics such as setting goals and study skills. Students can 
also register for courses early. 


'These data are based on a computation of beginning postsecondary students data from the U.S. Depart- 
ment of Education’s National Center for Education Statistics (NCES) using the NCES QuickStats website 
(http://nces.ed.gov/datalab/quickstats). 
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• Financial supports: Students receive a tuition waiver that covers any gap 
between financial aid and college tuition and fees. Students also receive free 
MetroCards for use on public transportation, contingent on participation in 
key program services, and free use of textbooks. 

MDRC, a nonprofit education and social policy research organization, evaluated 
ASAP’s effects on students’ academic outcomes, and its implementation and costs. The study 
targeted low-income students who needed one or two developmental courses at three CUNY 
community colleges: Borough of Manhattan, Kingsborough, and LaGuardia. This report shows 
that, compared with usual college services, ASAP substantially increased enrollment in college 
and credit accumulation, and nearly doubled graduation rates after three years. ASAP’s effects 
are the largest MDRC has found in more than a decade of research in higher education. 


Who Is in the Evaluation Sample? 

MDRC used a random assignment research design to evaluate the impacts (or effects) of ASAP 
on students’ academic outcomes over a three-year study period (or follow-up period), compared 
with usual services and courses at the colleges. The study targeted students who met the 
following eligibility criteria at the point of random assignment: had family income below 200 
percent of the federal poverty level or were eligible for a Pell Grant (or both), needed one or two 
developmental courses (to build math, reading, or writing skills), had previously earned 12 
credits or fewer, were New York City residents, were willing to attend college full time, and 
were in an ASAP-eligible major. Each eligible student who agreed to participate was assigned, 
at random, either to the program group, whose members had the opportunity to participate in 
ASAP, or to the control group, whose members had the opportunity to receive the usual college 
services. One group (or cohort) of students was assigned just before the spring 2010 semester 
and the other just before the fall 2010 semester. 

The 896 students in the sample completed a Baseline Information Form (BIF) just be- 
fore they were randomly assigned. Roughly two-thirds of the students in the research sample are 
women and most are relatively young. (Seventy-seven percent were 22 years of age or younger 
when they entered the study.) Reflecting the collective student body at the three colleges, the 
study sample is racially diverse, with no racial majority. The majority of sample members lived 
with their parents, were unmarried, and did not have children. 


What Are the Key Findings? 

Key findings from this report include the following: 
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• ASAP was well implemented during the period studied, and the differ- 
ence between ASAP and usual college services available to the study’s 
control group was substantial. 

ASAP is jointly administered by the CUNY Office of Academic Affairs (CUNY Cen- 
tral) and the participating community colleges. CUNY Central devised the program model and 
oversees the colleges’ implementation of the program. Each college has an ASAP team that 
operates the program and provides services to students. In addition, a CUNY Central ASAP 
team leads various internal evaluation efforts to assess the program’s effects and to improve the 
program. College ASAP staff record information about students’ contact with advisers and 
career and employment staff in a centralized data management system. Those data are analyzed 
monthly and the information is used to help manage and modify the program. College staff also 
track weekly attendance at tutoring and ASAP seminars and report that attendance information 
to CUNY each semester. Overall, during the period studied, the ASAP program operated with a 
high level of monitoring and assessment, with a focus on ongoing improvement. 

MDRC found that ASAP’s requirement to attend school full time was communicated 
and enforced, and its messages to take developmental courses early and graduate within three 
years were communicated effectively. The program provided much more intensive student 
services than typically available, particularly advising. The implementation of ASAP’s blocked 
or linked classes varied across the colleges, but most program group students took at least one 
course with a concentration of ASAP students. ASAP linked the receipt of monthly MetroCards 
to participation in key program services, such as advising. ASAP provided students free use of 
textbooks, and provided the tuition waiver to those whose need-based financial aid did not 
cover their tuition and fees. Table ES.l provides more detail about ASAP’s implementation and 
the differences between ASAP and the usual college services. 

• ASAP substantially improved students’ academic outcomes over three 
years, almost doubling graduation rates. 

This random assignment evaluation, like CUNY’s internal study of ASAP, 2 found that 
the program made a dramatic difference for students. Figure ES. 1 shows enrollment rates at any 
CUNY college for sample members over the three-year follow-up period. The colleges have 
main sessions, similar to traditional fall and spring semesters, followed by shorter intersessions; 


2 Donna Linderman and Zineta Kolenovic, Early Outcomes Report for City University’ of New York 
(CUNY) Accelerated Study in Associate Programs (ASAP) (New York: The City University of New York, 
2009); Donna Linderman and Zineta Kolenovic, Results Thus Far and the Road Ahead: A Follow-up Report 
on CUNY Accelerated Study in Associate Programs (ASAP) (New York: The City University of New York, 
2012 ). 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table ES.l 


Key Differences Between ASAP and Usual College Services 
Three-Year Impacts Report 


ASAP 

Usual College Services 

Reauirements 

• Full-time enrollment: Required 

• Taking developmental courses early: 

Encouraged consistently and strongly 

• Graduating within three years: Encouraged 
consistently and strongly 

and Messages 

• Full-time enrollment: Not required 

• Taking developmental courses early: 
Encouraged often but not strongly 

• Graduating within three years: Not typically 
encouraged 

Student 

• Advising: Student-to-adviser ratio between 
60:1 and 80:1; 95 percent of students met with 
an adviser during first year and students met 
with an adviser an average of 38 times in 
that period 

• Career services: 80 percent of students met 
with career and employment services staff 
during first year and students met with such 
staff an average of 9 times in that period 

• Tutoring: 74 percent of students received 
tutoring outside of class during first year and 
students met with a tutor an average of 24 
times in that period 

Services 

• Advising: Student-to-adviser ratio between 
600:1 and 1,500:1; 80 percent of students met 
with an adviser during first year and students 
met with an adviser an average of 6 times in 
that period 

• Career services: 29 percent of students met 
with career and employment services staff dur- 
ing first year and students met with such staff 
an average of 2 times in that period 

• Tutoring: 39 percent of students received 
tutoring outside of class during first year and 
students met with a tutor an average of 7 times 
in that period 

Course Ei 

• Blocked or linked courses: Available for first 
year; few students took complete block of 
courses, but most students took at least 1 class 
with a concentration of ASAP students 

• ASAP seminar: Most students took an ASAP 
seminar for 3 semesters 

nrollment 

• Blocked or linked courses: Available at 2 
colleges during first semester; participation in 
blocked or linked courses unknown 

• Support seminars: Some students took a 
freshman seminar or student success course 
during first year 

Financial 

• Tuition waiver: 3-1 1 percent of students 
received waiver, depending on semester 

• Free MetroCards: Most students received free 
MetroCards for use on public transportation, 
contingent on participation in the program 

• Free use of textbooks: Most or all students 
received textbooks 

Supports 

• Tuition waiver: Not available 

• Free MetroCards: Not available 

• Free use of textbooks: Not available 


SOURCE: MDRC field research data and MDRC student survey. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Figure ES.l 

Three- Year Enrollment at CUNY Colleges 



SOURCE: MDRC calculations from CUNY Institutional Research Database data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and research cohort. 

Enrollment is based on courses in which students are still enrolled as of the end of the add/drop 
period. 

"Main" represents the main session for the semester. "Inter" represents the intersession for the 
semester. 


the figure shows the semesters’ main sessions and intersessions separately. ASAP increased the 
proportion of students who enrolled in college during most sessions of the follow-up period. 
The effects are especially large during the intersessions. For example, during the intersession of 
the second semester, 54 percent of the program group enrolled, compared with only 29 percent 
of the control group. Because control group students enrolled at relatively low rates during 
intersessions, they left much room for improvement, and ASAP advisers encouraged students to 
take classes during intersessions to achieve full-time status and to continue accumulating 
credits. Not shown in the figure, ASAP increased full-time enrollment throughout the follow-up 
period. ASAP also substantially increased the average number of credits students earned. As 
Table ES.2 shows, program group members earned an average of approximately 48 credits over 
three years, about 9 credits more than their control group counterparts. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table ES.2 


Three-Year Summary of Credits Earned, Degrees Earned, 
and Enrollment in a Four-Year College 


Outcome 

Program 

Group 

Control 

Group 

Difference 

Total credits earned 

47.7 

39.0 

g 7 *** 

Earned a degree from any college (%) 

40.1 

21.8 

18.3 *** 

Enrolled in a 4-year college in semester 6 (%) 

25.1 

17.3 

7 g *** 

Sample size (total = 896) 

451 

445 



SOURCE: MDRC calculations from CUNY Institutional Research Database and National Student 
Clearinghouse data. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 


Most important, 40 percent of the program group students had received a degree by the 
end of the three-year follow-up period, compared with 22 percent of the control group — an 1 8 
percentage point impact. ASAP’s effects are the largest MDRC has found in any large-scale 
experimental study of a program in higher education. For example, among MDRC’s evaluations 
of community college refonns, the next largest increase in three-year graduation rates is 4 
percentage points. 3 ASAP’s effect on graduation is especially notable given that students 
needed some developmental courses when they entered the study. 

At the end of three years, ASAP had increased the proportion of students who trans- 
ferred to a four-year college. As Table ES.2 shows, 25 percent of the program group was 
enrolled in a four-year college in the last semester of the follow-up period, compared with 17 
percent of the control group. Whether enrollment itself leads more program group members to 
earn a bachelor’s degree is an open question, but the step of enrolling in a four-year college is a 
critical milestone. 


3 Reshma Patel, Lashawn Richburg-Hayes, Elijah de la Campa, and Timothy Rudd, Using Financial Aid to 
Promote Student Progress: Interim Findings from the Performance-Based Scholarship Demonstration (New 
York: MDRC, 2013). 
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The effects described above represent ASAP’s average impact across the full sample. 
ASAP had large positive effects for all of the subgroups of students examined in this report, 
including those defined by gender, receipt of high school diploma at baseline, and number of 
developmental courses needed at the start of the study. This finding provides evidence that 
ASAP was effective for a range of students, including some who tend to have lower overall 
success rates in higher education, such as men and students with multiple developmental needs. 

• At the three-year point, ASAP was found to be cost-effective; the cost 
per degree was lower in ASAP than in the control condition. 

A comprehensive analysis of ASAP’s costs estimated that the program’s services and 
benefits cost approximately $14,000 more per student than usual college services over the 
study’s three-year follow-up period. MDRC estimated that an additional $2,300 was spent on 
costs associated with program group members attempting more credits than control group 
members. Despite ASAP’s higher total cost (about $16,300, or 63 percent more than CUNY 
spent per student on usual college services), the cost per degree was lower because ASAP 
generated so many more graduates over the three-year follow-up period than did the usual 
college services. 


What Are the Implications of the Findings? 

Key implications of the findings include the following: 

• The findings in this report show that a comprehensive, long-term inter- 
vention can substantially boost students’ success. 

In contrast with many other reforms, ASAP targets multiple potential barriers to stu- 
dents’ success by providing an array of services and supports over three years. This approach 
allows ASAP to help a range of students with different barriers, including students with multi- 
ple barriers or different barriers over time. Despite ASAP’s incremental cost, the program has a 
lower cost per graduate than the usual college services because it boosted graduation rates so 
dramatically — at least at the three-year point. 

• Developmental education students’ outcomes can be markedly im- 
proved with the right package of supports, requirements, and messages 
— without changing what happens in the classroom. 

MDRC’s evaluation shows that ASAP was highly effective for students who needed 
one or two developmental courses. The higher education field has been struggling to develop 
initiatives to substantially help developmental education students, and ASAP has provided by 
far the most encouraging results of any community college reform that MDRC has yet evaluat- 
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ed. ASAP provides enhanced student services, including tutoring, and financial supports, but it 
does not change the curriculum or pedagogy in developmental education classrooms. There is 
still work to be done to improve what happens inside the community college classroom — and 
many refomis are being tried — but ASAP offers an alternative approach with great promise. 

• ASAP was designed to help a range of students, and this report presents 
evidence that it succeeded. 

As noted above, the majority of students in the evaluation sample were relatively young 
when they entered the study, lived at home with their parents, were unmarried, and did not have 
children. At the same time, however, the sample included a substantial number of nontraditional 
college students: 23 percent of the evaluation sample were 23 or older when they entered the 
study, 26 percent did not live with their parents, 31 percent were employed, 15 percent had at 
least one child, and at least 6 percent were married. 4 ASAP also generated positive effects for 
all of the subgroups of students examined. 

ASAP requires students to enroll full time. Nationwide, about 40 percent of community 
college students (roughly 2.8 million students) abend school full time. 5 Some higher education 
experts argue that many community college students simply cannot attend full time because of 
family obligations, work, or other issues. This study shows that ASAP boosted full-time 
enrollment, compared with usual college services — in other words, some students who would 
have otherwise attended college part time attended full time because of ASAP. It is unclear, 
however, what the program’s effects might be with a different target group, such as low-income 
parents, who might have more trouble enrolling full time. 


What Can Other Colleges Learn from ASAP? 

Exactly which program components drove ASAP’s effects is unknown. Ultimately, every- 
thing in ASAP’s comprehensive package of requirements, messages, services, and supports 
had the potential to affect students, and MDRC’s evaluation estimated the effect of that full 
package. The report, however, explores whether any particular aspects of the program may 
have been more or less important in changing students’ outcomes. This exploration, while 
much more speculative than the findings presented above, underscores the importance of 
several key features: 


4 l i I teen percent of the evaluation sample did not indicate their marital status on the BIF. 
5 American Association of Community Colleges’ Community College Enrollment website 
(http://www.aacc.nche.edu/AboutCC/Trends/Pages/enrollment.aspx). 
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• Requiring full-time enrollment in college while also providing an array 
of ongoing supports for students, such as enhanced advisement and fi- 
nancial supports, can yield substantial changes in enrollment and credit 
accumulation. 

ASAP’s full-time enrollment requirement, coupled with multiple supports to facilitate 
that enrollment, seem to be central to the program’s success. It is one thing to tell students to 
attend full time and hope that they can find the resources to do so; it is an entirely different thing 
to tell them to attend full time while also covering their tuition, books, and transportation, and 
providing an array of student services to support them in school. A survey of students about a 
year after the study started found that 90 percent of program group members believed that they 
had most or all of the services and supports that they needed to succeed in school. It is unknown 
exactly how much support is necessary to yield substantial effects on full-time enrollment, but it 
seems unlikely that such a requirement paired with far more limited financial and student 
service supports would be as fruitful. 

• Intersessions, perhaps especially in summer, provide good opportunities 
to increase enrollment in college and credit accumulation. 

ASAP’s impacts on enrollment were especially large in winter and summer interses- 
sions. Far fewer students receiving usual services typically enroll in winter and summer than do 
ASAP students, leaving much room for improvement. ASAP advisers encouraged students to 
enroll in intersessions both to achieve full-time status and to accelerate their progress through 
school, and ASAP’s tuition supports covered that enrollment. Notable enrollment increases in 
intersessions have been found in prior studies as well, including a study of a scholarship that 
was offered during the summer. 6 

• Requiring students to participate in key program components, monitor- 
ing participation, and providing a meaningful benefit to those who par- 
ticipate fully can markedly increase receipt of services. 

During this study’s follow-up period, students were required to see their adviser twice a 
month, meet with ASAP career and employment staff once a semester, and attend tutoring 
frequently if they were taking a developmental course or were on academic probation. ASAP 
linked meeting the participation requirements with receipt of a monthly MetroCard for use on 


6 See, for example, Reshma Patel and Timothy Rudd, Can Scholarships Alone Help Students Succeed? 
Lessons from Two New York City Community Colleges (New York: MDRC, 2012); Michael J. Weiss, 
Alexander Mayer, Dan Cullinan, Alyssa Ratledge, Colleen Sommo, and John Diamond, A Random Assign- 
ment Evaluation of Learning Communities at Kingsborough Community College: Seven Years Later (New 
York: MDRC, 2014). 
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public transportation. By the end of the follow-up period, each card cost $112 — a substantial 
cost for a low-income student. In a city where millions of people travel primarily or exclusively 
on public transportation, a MetroCard is a strong incentive and very likely increased students’ 
participation in ASAP services. As Table ES.l shows, the program group had far more contact 
with advisement, career services, and tutoring. 

To effectively link MetroCard receipt with use of student services, it was important to 
have information about that participation. Using a centralized data management system operat- 
ed by CUNY Central, college staff closely tracked students’ participation in advising and career 
and employment services. College staff also recorded attendance at tutoring every week. This 
tracking allowed the program staff to closely monitor students’ participation, adjust advisement 
as needed, and distribute MetroCards appropriately. 

• Monitoring program operations, with a focus on ongoing improvement, 
contributes to strong implementation. 

CUNY Central and the colleges also used the data tracking system to monitor program 
implementation, and conducted several internal evaluations. It seems likely that the high level of 
monitoring, with a focus on improvement, contributed to ASAP’s strong implementation and, in 
turn, to its positive effects for students. 

• Encouraging or requiring students to take developmental courses early 
in their time in college can hasten and increase completion of those 
requirements. 

Students in the program group moved through their developmental courses more quick- 
ly, and after three years many more program group members than control group members had 
completed their developmental education requirements. 


What’s Next? 

CUNY has been expanding ASAP, with a goal of serving over 13,000 students by fall 2016. 
While most ASAP features have remained the same, each ASAP adviser now works with no 
more than 150 students — more than during the evaluation but still far fewer than the number of 
students with whom advisers typically work. After the first semester, students are placed into 
one of three groups based on academic, personal resiliency, and program compliance criteria. 
The groups receive differentiated types of contact (individual, group, telephone, and e-mail) 
with a continued focus on maintaining strong relationships with their adviser. Tracking data 
from CUNY shows that the vast majority of students have maintained contact with their adviser 
at least once a month. 
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While ASAP’s three-year effects are notable, it will be necessary to continue tracking 
students’ outcomes to learn about the long-term effects. At the three-year point, ASAP may 
have caused students to graduate who would not have otherwise, accelerated graduation for 
students who would have graduated eventually, or both. Further research questions include the 
following: Do program group students continue to earn associate’s degrees at high rates? Do 
control group members begin to catch up? What are the long-term effects on receipt of bache- 
lor’s degrees? It is also valuable to know what happens to students in the labor market. A 
college degree is important in part because of its potential to benefit its recipient in the form of 
better job opportunities. 

ASAP has received much attention in the field of higher education, and many colleges 
have begun exploring whether they might adopt a similar model. Indeed, ASAP is a highly 
promising program that warrants testing in other settings. It would be useful to understand 
whether ASAP can be implemented by other colleges, in different contexts, with different 
students, and yield substantial effects. To help answer those questions, MDRC is working with 
CUNY to develop evaluations of ASAP replications at interested colleges. 

*** 

ASAP’s effects are the largest MDRC has found in any of its evaluations of community 
college reforms. The evaluation found that CUNY and three of its community colleges success- 
fully implemented a program that generated large, meaningful impacts for low-income students 
with developmental education needs in an urban setting. With an investment in the right 
combination of services, requirements, and messages, community college students can succeed 
at far higher rates. The ASAP model offers a highly promising strategy to markedly accelerate 
students’ progress, increase graduation rates, and build human capital among educationally and 
economically disadvantaged populations. 
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Chapter 1 

Introduction 


Postsecondary education is widely seen as the best way for low-income individuals to enter 
well-paying careers and move into the middle class. Yet despite the great strides made in the 
last several decades in opening up college education to women, students of color, and low- 
income students, graduation rates remain stubbornly low, particularly for the latter two groups. 
This problem is especially pronounced in community colleges, the primary postsecondary 
education providers for low-income students in the United States. Among full-time, first-time, 
degree-seeking students entering public two-year colleges, only 20 percent graduate with a 
degree within three years, or 150 percent of what is considered “normal” for an associate’s 
degree. 1 By five years, that number increases to only 35 percent, while 45 percent of students 
have not earned a degree and are no longer enrolled in college at all. 2 

Low-income students in particular face immense challenges when they enter communi- 
ty college. Financial aid may not cover all of students’ cost of attendance, leaving them unable 
to afford tuition, books, or transportation. 3 To cope, they may take on more work hours or enroll 
in school only part time, both of which correlate with reduced academic success. 4 Many 
students juggle multiple priorities, including family and work, alongside school. Low-income 
students and students of color may be especially hard hit, as they may be less likely to under- 
stand how to navigate college, particularly those who are first in their families to attend. 5 

In addition to economic and social barriers to success, community college students face 
academic barriers. The majority of first-time community college students are referred to 
developmental (or remedial) education, meaning that they must take basic courses in math and 
English before they are considered ready to take college-level courses. Not only do these 
students’ graduation rates lag behind their peers’, but the majority do not even complete the 
developmental course sequences to which they are referred. 6 

College administrators and policymakers across the country are looking for new pro- 
grams and policies that can improve degree completion rates and developmental education 
success rates. One promising example is Accelerated Study in Associate Programs (ASAP), 


'Snyder and Dillow (2013). 

2 Green and Radwin (2012). 

3 The Institute for College Access and Success (2009). 

4 The Advisory Committee on Student Financial Assistance (2001). 
5 Palmer, Davis, Moore, and Hilton (2010). 

6 Bailey, Jeong, and Cho (2010). 
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created by the City University of New York (CUNY). Launched in 2007 with funding from the 
New York City Center for Economic Opportunity, ASAP is one of the most ambitious efforts in 
the country to improve the success rates of low-income postsecondary students. As of spring 
2014, ASAP had served more than 6,400 students. 

ASAP is both an opportunity and an obligation for students. ASAP requires students to 
attend college full time and provides a rich array of supports that allows them to do so. Unlike 
many programs, ASAP is designed to address multiple potential barriers to student success 
simultaneously and to address them over three full years. ASAP’s goal is to accelerate students’ 
academic progress and graduate at least 50 percent of participating students within three years, 
far exceeding the national average. The program includes the following key components: 

• Requirements and messages: Students are required to attend college full 
time during the fall and spring semesters and are encouraged to complete de- 
velopmental education early and to graduate within three years. 

• Student services: Students receive comprehensive advisement from an 
ASAP-dedicated adviser with a small caseload, career infomiation from an 
ASAP-dedicated career and employment services staff member, and ASAP- 
dedicated tutoring services separate from the usual college tutoring services. 

• Course enrollment: Students enroll in blocked or linked courses in their first 
year. Students also enroll in an ASAP seminar covering topics such as goal- 
setting, study skills, and academic planning. Students can register for courses 
early so that they can create convenient schedules and get seats in the courses 
they need. 

• Financial supports: Students receive a tuition waiver that fills any gap be- 
tween financial aid and school tuition and fees. Students also receive free use 
of textbooks and free MefioCards for use on public transportation, contingent 
on participation in the program. 

MDRC, a nonprofit, nonpartisan education and social policy research organization, 
conducted a random assignment evaluation of ASAP, examining its implementation, its effects 
on students’ academic outcomes, and its costs. For this study, ASAP targeted low-income 
students who needed one or two developmental courses and who were willing to attend school 
full time. While ASAP operates at six of CUNY’s community colleges, only three participated 
in this evaluation: Borough of Manhattan Community College, Kingsborough Community 
College, and LaGuardia Community College. The study began in 2009. This report presents 
three years of results from the study, including the largest estimated impacts on credit accumu- 
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lation and graduation rates of any of the higher education programs that MDRC has evaluated: a 
near doubling of graduation rates after three years. 


ASAP Context Within CUNY 

CUNY launched ASAP in 2007 to address the low rates of degree completion at its community 
colleges. The program initially targeted college-ready students — that is, students with no 
developmental education needs. However, in 2009, the program expanded to include students 
who needed some developmental education courses, in the hope that they, too, would benefit. 
Although the trend across the country has been to address the needs of developmental education 
students within the classroom, with changes to content and pedagogy of developmental courses 
paramount, CUNY was confident that ASAP could serve students needing developmental 
coursework just as it did college-ready students. 

CUNY’s leaders developed ASAP to address what they perceived as the major barriers 
affecting their community college students. Rather than address issues piecemeal, they designed 
a package of wraparound services that could address multiple issues over three full years. The 
majority of the components included in ASAP are based on research into what helps and 
hinders student success in college. However, unlike many postsecondary interventions that are 
short term and designed to address just one or a few barriers to graduation — and as a result, 
produce modest or short-term effects — ASAP lasts a full three years and addresses multiple 
barriers to completion. 

First, students must enroll in school full time in order to receive ASAP’s complete 
package of benefits. Full-time attendance correlates strongly with academic achievement. 7 In 
addition, ASAP students are encouraged to take their developmental courses quickly, starting in 
their first semester in college. While this has not been studied rigorously, it is a popular and 
widespread practice that many researchers and postsecondary institutions believe helps students 
succeed academically. 8 

ASAP requires students to attend academic advising, a key student service that can help 
students address topics such as which classes to take, balancing school with other responsibili- 
ties, accessing other student services, interacting with professors, and staying on track to graduate 
on time. This may be especially important in community colleges, where students are less likely 
to have the familial and social resources to help them navigate the path to graduation. Advising is 
intended to mediate these issues for students, leading them to make better decisions and solve 


7 Adelman (2006). 

s Long and Boatman (2013); Rutschow and Schneider (2011); Attewell, Lavin, Domina, and Levey 
(2006). 
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problems before they drop out of school and increasing persistence and success. An experimental 
evaluation of an enhanced advising program conducted by MDRC found that enhanced advising 
increased students’ persistence and the number of credits earned in the short term, though the 
effects faded over time . 9 A more recent experimental evaluation of individualized student 
coaching, similar to academic advising, found positive effects on persistence . 10 Continuing to 
provide advising throughout students’ time in college may sustain these effects long tenn. 

ASAP students must also attend tutoring while they take their developmental courses 
and if they are on academic probation. Tutoring is intended to help underprepared students by 
providing additional instruction, reinforcement, and support. Best practices in the field include 
offering multiple types of academic support, from one-on-one tutoring to computerized supple- 
mental instruction . 11 MDRC evaluated a program that offered students incentives to visit a 
tutoring center as part of a larger intervention and found modest positive results on academic 
perfonnance in the targeted course sequence . 12 

ASAP provides blocked and linked courses for students in their first year, the goals of 
which are to enroll ASAP students together in the same courses so that they can meet and 
support one another and to give program students convenient schedules so they can make the 
most of their time on campus. While this component does not reach the level of a classical 
learning community , 11 it is designed to provide some similar benefits, such as better acclimation 
to the college environment and the fomiation of meaningful bonds with fellow students. These 
benefits may be especially important in community colleges, since students commute rather than 
live on campus and may spend little time at school outside of class . 14 Experimental evaluations 
of learning communities conducted by MDRC have found mixed results, with some learning 
communities producing statistically significant 15 impacts on persistence and credit accumulation 
and others producing no statistically significant impacts . 16 However, there is evidence in the 
literature that strengthening students’ academic and social bonds in college leads them to persist 
from semester to semester . 17 


9 Scrivener and Weiss (2009). 

10 Bettinger and Baker (2013). 

"Fullmer (2012); Rheinheimer, Grace-Odeleye, Francois, and Kusorgbor (2010); Perin (2004). 

12 Sommo et al. (2014). 

13 ln learning communities, small cohorts of students are placed together in two or more courses for one 
semester, usually in the freshman year. The courses are linked by a common theme and are taught by a team of 
instructors who collaborate with each other to develop the syllabi and assignments. 

14 Taylor, Moore, MacGregor, and Lindblad (2003); Adelman (2005). 

1 5 A statistically significant impact is one that is unlikely to have arisen by chance and can be attributed to 
the intervention with a high degree of confidence. 

16 Weiss, Visher, Weissman, and Wathington (forthcoming). 

"Person, Rosenbaum, and Deil-Amen (2006); Deil-Amen (2011). 
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The same may be said about the ASAP seminar. In addition to providing social support, 
student success courses such as the ASAP seminar are intended to help students by teaching soft 
skills that are key to success, including time management and study skills. Success courses may 
be especially helpful for students who do not know how to navigate the college environment. 
MDRC conducted an evaluation of a success course aimed at students with developmental 
education needs. It found that the success course had a positive impact on students’ self- 
reported interdependence , 18 self-awareness, emotional intelligence, and engagement in college, 
among students with low levels of these attributes. However, the course did not have statistical- 
ly significant effects on students’ academic achievement . 19 

ASAP also provides financial supports to assist low-income students with costs other 
than tuition and fees. While nearly all ASAP students receive federal or state financial aid, 
many might still be unable to pay for textbooks and transportation to class. By providing low- 
income students with funds to fill the gap between government financial aid and total tuition 
and fees and to cover expenses such as use of textbooks and transportation, ASAP is designed 
to allow students to work fewer hours and experience reduced financial stress. These benefits, in 
turn, should allow them to focus more on their studies. Many researchers have evaluated 
financial aid interventions that have produced promising results, including positive impacts on 
credits earned and graduation . 20 

ASAP is unique in that it brings all of these components together in a single, compre- 
hensive program — without getting inside the classroom. By offering students an array of 
services, ASAP seeks to address the many barriers — academic, financial, and personal — that 
can impede students from succeeding in college. Taken together, the components of ASAP seek 
to effect positive academic outcomes for students. In the short tenn, students will progress 
through developmental education, earn more credits, and persist semester to semester. Long 
tenn, students will earn a degree and transfer to a four-year institution to continue their higher 
education and, as a result, achieve better employment and earnings outcomes. 

Notably, ASAP does not alter what takes place inside the classroom. ASAP students 
take the same academic courses with the same faculty and cover the same material as non- 
ASAP students. Unlike many refonns in developmental education that change practices inside 
the classroom, ASAP proposes that student outcomes can be improved by changing the college 
context outside the classroom. 


^Interdependence is defined as building mutually supportive relationships that foster the achievement 
of goals. 

19 Rutschow, Cullinan, and Welbeck (2012). 

20 Patel, Richburg-Hayes, de la Campa, and Rudd (2013); Goldrick-Rab, Harris, Kelchen, and Benson 
( 2012 ). 
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MDRC’s Evaluation of ASAP 

CUNY conducted extensive internal evaluations of ASAP to track students’ outcomes and 
continually improve the program. CUNY published two reports on ASAP comparing ASAP 
students’ outcomes with a constructed comparison group. 21 These reports found that students in 
ASAP were more likely than the constructed comparison group to complete developmental 
education, persist from semester to semester, enroll full time, and earn associate’s degrees 
within three years. CUNY also commissioned a study to investigate the cost-effectiveness of 
ASAP and the costs and benefits of the program. 22 This study found that ASAP was highly 
cost-effective in tenns of graduating students. While it cost more to run ASAP, the increase in 
the number of students receiving degrees outpaced the additional cost. 

In 2009, CUNY commissioned MDRC to conduct an external, experimental evaluation 
of ASAP. For the evaluation, ASAP exclusively targeted students who needed developmental 
education. MDRC used a random assignment research design to estimate the effects of ASAP 
on students’ academic outcomes over a three-year period, compared with the effects of the usual 
college services. Widely considered to be the gold standard in social science research, random 
assignment creates two groups that are similar with respect to both observable characteristics, 
such as race, developmental need, and age, and unobservable characteristics, such as tenacity 
and motivation. Students in one group are assigned to receive the program’s services, while 
students in the other group receive the usual college services. Because the groups are similar at 
the outset, differences in outcomes found later provided an unbiased estimate of the causal 
effect of the program. 

The evaluation examined ASAP’s estimated effects on students’ academic progress and 
completion, including persistence, credit accumulation, degree receipt, and transfer to four-year 
universities. The study also looked at ASAP’s impacts on subgroups of students. 

In addition to the impact analysis, MDRC’s evaluation explored how ASAP was im- 
plemented at the three colleges in the study and how the program services that ASAP offers 
compared with the usual college services. The study also examined participation in various 
services by program and control group members as reported on a student survey and the nature 
of the treatment contrast, or the difference between the services received by the program group 
and the services received by the control group. Although the evaluation cannot determine which 
components of ASAP matter most, the implementation and survey findings may help shed light 
on that issue. 


21 Linderman and Kolenovic (2012); Linderman and Kolenovic (2009). 
22 Levin and Garcia (2012); Levin and Garcia (2013). 
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Finally, MDRC examined the costs of ASAP. Because ASAP provides many services, it 
is more expensive than most other programs MDRC has studied. The cost-effectiveness analysis, 
however, determines whether the positive impacts on student outcomes justify the cost. 


Summary of Findings 

MDRC has released two prior publications on the evaluation of ASAP. 23 A 2012 report covered 
early findings from the program, including positive first-year impacts on credits earned and 
developmental course completion. In 2013, MDRC released a brief presenting two-year 
findings on ASAP, showing that it increased students’ likelihood of enrolling each semester, 
including summer and winter intersessions; increased students’ credit accumulation, including 
both developmental and college-level credits; and increased students’ graduation rates by nearly 
6 percentage points after two years. 

This report follows up on those promising findings by carrying the analysis through to a 
full three years. It finds increasingly large impacts on credit accumulation and graduation as 
students reached three years, the benchmark established by CUNY for completing the program. 

Chapter 2 of this report describes the study, background, and data sources in more 
depth. Chapter 3 provides an in-depth implementation analysis of the ASAP components as 
well as presents survey results showing the treatment contrast. Chapter 4 assesses ASAP’s 
impacts on students’ academic outcomes. Chapter 5 presents a cost-effectiveness analysis of the 
ASAP program. Chapter 6 concludes the report with a discussion of future directions for ASAP 
and its evaluation. 


23 Scrivener, Weiss, and Sommo (2012); Scrivener and Weiss (2013). 
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Chapter 2 

Sites, Evaluation Sample, and Data Sources 


This chapter begins by briefly describing the three City University of New York (CUNY) 
community colleges that participated in the evaluation. It then explains the random assignment 
process that was used at the colleges to build the sample for the evaluation and describes the 
characteristics of the sample members when they entered the study. The final section of the 
chapter presents the data sources used in this report. 


Colleges in the Evaluation 

One of the largest public university systems in the country, CUNY serves more than half a 
million students annually. It consists of 24 institutions across the five boroughs of New York 
City, including four-year colleges, graduate and professional schools, and seven community 
colleges. CUNY’s Office of Academic Affairs (CUNY Central) oversees the academic policies 
of the colleges and is responsible for the development and implementation of special multi- 
college programs, such as ASAP. 1 

ASAP operates at the six community colleges that existed in 2007 when the program 
was launched. 2 Three of those colleges participated in the ASAP evaluation: Borough of 
Manhattan Community College (BMCC), Kingsborough Community College (KCC), and 
LaGuardia Community College (LGCC). CUNY and MDRC selected the three colleges 
primarily based on administrators’ willingness to participate in an evaluation and work with 
MDRC to develop and implement a process to randomly assign students, and their capacity to 
reach the desired sample size goals. They are the largest of CUNY’s community colleges. 

BMCC is located in lower Manhattan, and the school served about 21,000 students 
when the study started. KCC is located in Brooklyn and LGCC is located in Queens; both 
colleges enrolled about 15,000 students at the start of the study. The majority of students at each 
college attend school full time (about two-thirds at BMCC and LGCC and three-fourths at 
KCC). All three colleges offer a wide range of associate’s degree programs that prepare 
students to transfer to four-year colleges or enter professional careers. 


'See the CUNY website at www.cuny.edu. 

2 The seventh community college, Guttman Community College, opened in 2012. 
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Random Assignment Process and the Sample Members 

MDRC’s evaluation targeted students at BMCC, KCC, and LGCC who met the following 
eligibility criteria when they entered the study: 

• Family income below 200 percent of the federal poverty level or eligible for 
a Pell Grant (or both) 

• In need of one or two developmental courses based on CUNY Assessment 
Tests 3 

• New student or continuing student who had earned 12 credits or fewer and 
had at least a 2.0 grade point average 

• New York City resident 

• Said they were willing to attend college full time 

• In an ASAP-eligible major (the colleges excluded a few majors that have re- 
quirements that make graduating quickly difficult) 4 

The eligibility criteria for the evaluation mirror those for ASAP when the study started, 
with two exceptions. First, ASAP accepted some college-ready students who did not need any 
developmental courses, but the evaluation did not. MDRC chose to focus on the effects of 
ASAP for students with developmental education needs. In fall 2009, the semester before the 
evaluation began, 77 percent of the students who began in ASAP across the six colleges needed 
one developmental course or more and 23 percent did not. 5 During the semesters of sample 
intake, the ASAP programs at the three colleges in the evaluation enrolled only students with 
developmental education needs, while the three colleges that were not in the evaluation contin- 
ued to enroll some college-ready students. 

Second, ASAP serves some students who are undocumented immigrants. These stu- 
dents are ineligible to receive federal or state financial aid and thus ineligible for the ASAP 
tuition waiver. The colleges in the evaluation continued to serve undocumented immigrants 
during the sample intake period, but because the evaluation was designed to test the effects of 


3 In other words, students need one math course, one English course, two math courses, two English cours- 
es, or one math course and one English course. 

4 The excluded majors at the time were: at BMCC, Allied Elealth Sciences, Pre-Clinical Nursing, Forensic 
Science, and Engineering Science; at KCC, Nursing; and at LGCC, Allied Health Sciences and Engineering 
Science. 

5 Linderman and Kolenovic (2012). 
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the full package of ASAP services including the tuition waiver, these students were not included 
in the evaluation sample. 6 

ASAP staff at BMCC, KCC, and LGCC invited students who met the eligibility criteria 
to participate in the evaluation through letters, e-mails, and phone calls. Students who attended 
an intake session on campus, during which time staff described the ASAP program and evalua- 
tion, and who agreed to take part in the study completed an Infomied Consent Form and a 
Baseline Information Form (BIF) containing questions about their background. After complet- 
ing the fonns, each student was randomly assigned (using a computer program at MDRC) either 
to the program group, whose members had the opportunity to participate in ASAP, or to the 
control group, whose members had the opportunity to receive the usual college services. As 
compensation for their time, students who were randomly assigned to either group received a 
one-week MetroCard for use on public transportation. 7 

Two groups (or cohorts) of students were randomly assigned for the MDRC evaluation: 
one just before the spring 2010 semester and the other just before the fall 2010 semester. 
(BMCC and KCC assigned students before both semesters; LGCC assigned students only 
before the fall 2010 semester.) A total of 896 students are in the sample — 451 in the program 
group and 445 in the control group. 

The right-most column of Table 2.1 shows selected characteristics of the evaluation 
sample from the BIF and data from CUNY. As the table shows, 62 percent are women and 
students’ average age was 21.5 when they entered the study. Although the majority of the 
students in the sample were relatively young when they were randomly assigned, almost one- 
fourth (23 percent) were 23 years of age or older. The sample is racially and ethnically diverse: 
44 percent are Hispanic, 34 percent black, 10 percent white, and 8 percent Asian or Pacific 
Islander. The vast majority of the sample members (88 percent) were receiving a Pell Grant. 
According to available data, 60 percent of the sample needed developmental instruction in one 
subject (math, reading, or writing) and 27 percent needed instruction in two subjects. 8 Appendix 
Table A.1 shows some additional characteristics of the students in the sample from the BIF. As 
that table shows, most sample members were unmarried and did not have any children when 
they entered the study, and most lived with their parents. About one-third reported that they 


6 According to CUNY, 19 undocumented immigrants entered the ASAP program at the three colleges in 
the study during the sample intake period. 

7 The MetroCard covered unlimited subway and bus fares for seven days and cost $27. 

developmental education need is shown in Table 2.1 as “unknown” for students who did not have 
CUNY Assessment Test data in all subject areas. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 2.1 


Selected Characteristics of MDRC ASAP Evaluation Sample 
Compared with the College Population 

Three- Year Impacts Report 


Characteristic 

Freshman 

Population 3 

Target 

Population 11 

Evaluation 

Sample 

Total students 

35,163 

8,520 

896 

Gender (%) 

Male 

45.3 

44.2 

37.9 

Female 

54.7 

55.8 

62.1 

Age (%) 

18 or younger 

30.6 

28.1 

39.1 

19 

18.8 

20.5 

18.1 

20 to 22 

25.7 

28.9 

19.4 

23 to 29 

16.2 

15.9 

14.5 

30 or older 

8.6 

6.7 

8.9 

Average age (years) 

22.1 

21.7 

21.5 

Race/ethnicity (%) 

Hispanic 

36.7 

40.2 

43.6 

White 

16.8 

12.0 

10.0 

Black 

31.4 

34.7 

34.3 

Asian or Pacific Islander 

14.7 

12.8 

7.5 

Other 0 

0.3 

0.2 

4.6 

Student's status (%) 

Incoming freshman 

45.9 

45.3 

60.0 

Returning student 

42.6 

45.8 

33.5 

Transfer student 

11.6 

9.0 

6.5 

Pell status' 1 

Received a Pell Grant 

65.7 

96.7 

87.5 

Did not receive a Pell Grant 

15.7 

3.3 

6.5 

Developmental need 0 (%) 

College-ready 

24.8 

NA 

1.9 

1 subject 

28.7 

55.1 

59.9 

2 subjects 

21.4 

44.9 

26.8 

3 subjects 

13.7 

NA 

0.3 

Unknown 1 

11.4 

NA 

11.0 


(continued) 
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Table 2.1 (continued) 


SOURCES: MDRC calculations from CUNY Institutional Research Database (IRDB), Integrated 
Postsecondary Education Data System (IPEDS), and Baseline Information Form (BIF) data. 

NOTES: Distributions may not add to 100 percent because of rounding. 

“This category includes students seeking an associate's degree at Borough of Manhattan 
Community College (BMCC), Kingsborough Community College (KCC), and LaGuardia Community 
College (LGCC) who had 12 or fewer credits at the start of the spring 2010 or fall 2010. 

b This category is an approximation of the target group for the evaluation. It includes students 
seeking an associate's degree at BMCC, KCC, and LGCC who had 12 or fewer credits at the start of 
the spring 2010 or fall 2010 term, were proficient in one or two subjects, were enrolled in ASAP 
approved majors, were NYC residents, and had an income below 200 percent of the federal poverty 
guideline or were in receipt of a Pell Grant. 

c For the first two columns, the "Other" category represents students who selected American 
Indian/Native Alaskan on CUNY documents. For the last column, the "Other" category represents 
students who selected Native American, Alaska Native, or Other on the BIF. 

d Categories do not add to 100 percent as students who did not complete a FAFSA are not included. 
Estimates for percentages of college-ready students, students needing developmental education in 
three subjects, and students with unknown developmental needs are not given for target population 
students, as the target population included only students with one or two developmental needs. 

^Developmental need is unknown for students without CUNY Assessment Test data in all subject 
areas. 


were working at the point of random assignment. Almost all sample members said they planned 
to earn a degree beyond an associate’s degree. 9 

Table 2.1 shows selected characteristics for two other groups of students: (1) in the first 
column, freshmen and relatively new students (those who had earned 12 or fewer credits); and 
(2) in the second column, students who met the eligibility criteria for the study (based on 
available data). This evaluation estimates ASAP’s effects for the evaluation sample, who, as 
noted above, agreed to participate in the study. When considering whether ASAP’s effects on 
the evaluation sample are likely to generalize to the broader target population, it is important to 
consider how the characteristics of the students in the evaluation compare with the characteris- 
tics of students in the target population — that is, whether the evaluation sample is representa- 
tive of the target population. Overall, the characteristics presented in Table 2.1 suggest that the 
evaluation sample looks relatively similar to the target population. Apart from the key ASAP 
and evaluation eligibility criteria of receiving a Pell Grant and needing one or two developmen- 


9 A table in a prior report presented baseline characteristics for program group and control group mem- 
bers separately (Appendix Table 1 in Scrivener, Weiss, and Sommo, 2012). The two groups’ characteristics 
were very similar, suggesting that the random assignment process succeeded in creating two similar groups 
of students. 
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tal courses, the characteristics of the target population and those of freshmen or relatively new 
students are relatively similar. 

Reflecting the eligibility criteria, most or all of the target population and evaluation 
sample needed developmental work in one or two subjects. 10 Based on the available data, half 
of the freshman population needed developmental instruction in one or two subjects; most of 
the others needed either no developmental courses or developmental instruction in three 
subjects. At the three colleges participating in the study, ASAP targeted the roughly half of 
students in the middle of the range of basic skills. 


Data Sources 

MDRC’s evaluation of ASAP relied on several data sources. The analyses in this report are 
based primarily on the data sources described below. 

• Baseline data: The BIF included demographic and other background infor- 
mation. BIF data are used above to help describe the students in the evalua- 
tion sample. In Chapter 4, BIF data are used to identify subgroups of sample 
members in order to estimate ASAP’s effects for different types of students. 

• CUNY Assessment Test data: Students at CUNY’s community colleges are 
required to take the CUNY assessment tests in reading, writing, and math be- 
fore they begin classes. Test score data are used in this chapter to help de- 
scribe students when they entered the study and in Chapter 4 to estimate stu- 
dents’ progress through developmental education and to define subgroups of 
students. 

• Field research: Throughout the course of the evaluation, MDRC staff con- 
ducted field research visits to the three participating colleges to interview the 
administrators and staff involved in ASAP. The interviews provided detailed 


l0 Appendix Table A.2 shows more information on the sample’s developmental education needs at base- 
line. Based on CUNY’s Institutional Research Database (IRDB) data, 39 percent of the students in the 
sample needed one course and 51 percent needed two courses. Two percent of the students in the sample did 
not need any developmental courses, and 8 percent needed three or more. One college implemented the 
eligibility criteria differently from the others and accepted students who needed multiple math courses even 
if it meant they needed more than two courses; that college accounts for the majority of students whom 
MDRC identified as needing three or more developmental courses at baseline (three-fourths of the 8 
percent). Nearly all other students identified by MDRC as needing no courses or three courses or more 
actually needed one or two courses according to CUNY analysis using information other than the IRDB 
data. Appendix Table A.2 also shows that most students in the sample needed remediation in math; 17 
percent needed remediation only in English. 
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information on the operation of the program. During one visit in 2011, 
MDRC staff also interviewed selected administrators and staff about the usual 
college services that were available to the students in the study’s control 
group. In between field visits, MDRC conducted periodic phone calls with the 
ASAP leadership at CUNY Central and the colleges to discuss program oper- 
ations. Information from all these interviews is used in Chapter 3 to describe 
ASAP and the key differences between ASAP and the control enviromnent. 

• MDRC student survey: A survey was administered to all sample members 
approximately one year after they were randomly assigned. The survey was 
initially administered on the Internet; students who did not respond were 
contacted by phone. In the end, 83 percent of the students in the evaluation 
sample completed the survey — 85 percent of the program group members 
and 81 percent of the control group members. As is described in Appendix 
B, the survey results are likely representative of the full sample. The survey 
covered topics such as sample members’ participation in and experiences 
with student services, expectations and engagement in college, employment, 
and financial aid and other financial issues. Findings from the survey are 
used in Chapter 3 to help describe the treatment contrast between ASAP and 
the usual college services. 

• CUNY ASAP tracking data: ASAP staff at the colleges entered infor- 
mation on students’ contact with advisers and career and employment spe- 
cialists into a centralized data management system, for periodic analysis by 
CUNY. CUNY provided MDRC with summary findings from their analysis, 
covering the three years after random assignment. This information is used in 
Chapter 3. 

• Data on ASAP tuition waiver: CUNY ASAP staff analyzed data on pro- 
gram group students’ receipt of the ASAP tuition waiver for the three years 
after random assignment for all sample members. This infonnation is used in 
Chapter 3. 

• CUNY student transcript data: CUNY provided MDRC with infonnation 
on students’ course taking and degree receipt at the three colleges in the 
study and all the other colleges in the CUNY system. Transcript data from 
CUNY’s Institutional Research Database (IRDB) are available for three 
years after random assignment for all sample members. These data are used 
in Chapter 4 to provide a detailed look at the impacts of ASAP on students’ 
academic outcomes, and in Chapter 5 to examine the cost-effectiveness of 
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the program. Transcript data prior to random assignment are used in one 
analysis in Chapter 4. 

• Student records from the National Student Clearinghouse (NSC): The 

NSC, a nonprofit organization, collects and distributes enrollment, degree, 
and certificate data from more than 3,500 colleges that combined enroll more 
than 98 percent of the nation’s college students. 11 NSC data are available for 
three years for all sample members. The data are used in Chapter 4 to exam- 
ine enrollment and degree receipt at non-CUNY colleges and in Chapter 5 to 
examine cost-effectiveness. 

• Cost data: The CUNY Central ASAP office provided expenditure data for 
the three colleges that were part of the study. To help estimate control group 
costs, MDRC obtained operating budgets and annual credit hours attempted 
for the colleges from the Integrated Postsecondary Education Data System 
(IPEDS). IPEDS is a system of interrelated surveys conducted annually by 
the U.S. Department of Education’s National Center for Education Statistics 
that gathers infonnation from every college, university, and technical and 
vocational institution participating in federal student financial aid programs. 


1 'See the National Student Clearinghouse’s website (www.studentclearinghouse.org/about). 
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Chapter 3 


How Was ASAP Implemented and How Does 
ASAP Compare with Usual College Services? 


Whereas Chapter 1 presents the ASAP model as it was designed or intended, this chapter 
explains how ASAP was implemented over the three-year study period (or follow-up period) at 
the three City University of New York (CUNY) colleges that participated in the evaluation: 
Borough of Manhattan Community College, Kingsborough Community College, and LaGuar- 
dia Community College. The follow-up period was spring 2010 through summer 2013. Recall 
that students in ASAP can receive program services for up to three years. 1 Although the 
program for the most part was implemented similarly at the three colleges, this chapter high- 
lights notable differences. The chapter also describes the key differences between the experi- 
ences of the program group and control group students — the service contrast — that is at the 
heart of the study’s random assignment design. These differing experiences are what caused the 
effects on academic outcomes described in Chapter 4. 

The chapter is divided into several sections. In the sections that cover the program and 
control environment, ASAP is presented first, followed by the services available to the control 
group and the service contrast. The findings in this chapter are based on a few key data sources: 
MDRC field research, analysis from CUNY on program group members’ participation in key 
ASAP activities and tuition waiver receipt, and a survey of program group and control group 
students about one year after they entered the study. See Chapter 2 for more details on the data 
sources. 


Summary of Findings 

ASAP was implemented largely as designed, with some variation by college, as is allowed for 
in the program model. The program provided students with a wide array of services over a 
three-year period, and effectively communicated various requirements and other messages. 
Overall, the difference between ASAP and the usual services at the colleges was substantial. 
Table 3.1 describes the key differences. 

Although Table 3.1 and most of this chapter discuss ASAP component by component, 
ASAP is really a package of services and supports. The research can neither determine the 
interactive or complementary effects of the components, nor disentangle the effects of each 

'For the most part, this chapter uses past tense to describe program implementation, although much of 
what is discussed may be true of the current program. The program model is discussed in the present tense. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 3.1 


Key Differences Between ASAP and Usual College Services 
Three-Year Impacts Report 


ASAP 

Usual College Services 

Requirements and Messages 

• Full-time enrollment: Required 

• Full-time enrollment: Not required 

• Taking developmental courses early: 

• Taking developmental courses early: 

Encouraged consistently and strongly 

Encouraged often but not strongly 

• Graduating within three years: Encouraged 

• Graduating within three years: Not typically 

consistently and strongly 

encouraged 

Student Services 

• Advising: Student-to-adviser ratio between 

• Advising: Student-to-adviser ratio between 

60:1 and 80:1; 95 percent of students met with 

600:1 and 1,500:1; 80 percent of students met 

an adviser during first year and students met 

with an adviser during first year and students 

with an adviser an average of 38 times in that 

met with an adviser an average of 6 times in 

period 

that period 

• Career Services: 80 percent of students met 

• Career Services: 29 percent of students met 

with career and employment services staff dur- 

with career and employment services staff dur- 

ing first year and students met with such staff 

ing first year and students met with such staff 

an average of 9 times in that period 

an average of 2 times in that period 

• Tutoring: 74 percent of students received 

• Tutoring: 39 percent of students received 

tutoring outside of class during first year and 

tutoring outside of class during first year and 

students met with a tutor an average of 24 

students met with a tutor an average of 7 times 

times in that period 

in that period 

Course Enrollment 

• Blocked or linked courses: Available for first 

• Blocked or linked courses: Available at 2 

year; few students took complete block of 

colleges during first semester; participation in 

courses, but most students took at least 1 class 

blocked or linked courses unknown 

with a concentration of ASAP students 

• Support seminars: Some students took a 

• ASAP seminar: Most students took an ASAP 

freshmen seminar or student success course 

seminar for 3 semesters 

during first year 

Financial Supports 

• Tuition waiver: 3-1 1 percent of students 

• Tuition waiver: Not available 

received waiver, depending on semester 

• Free MetroCards: Not available 

• Free MetroCards: Most students received free 

• Free use of textbooks: Not available 

MetroCards for use on public transportation, 


contingent on participation in the program 


• Free use of textbooks: Most or all students 


received textbooks 



SOURCE: MDRC field research data and MDRC student survey. 
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individual component. Such a rich model can help a wide range of students, since different 
students face different barriers and many students face multiple barriers. 


ASAP Staffing, Administration, and Internal Evaluation 

This section briefly describes how ASAP was staffed and administered during the study period. 
(The report does not explicitly address the staffing or administration at the colleges outside of 
ASAP.) The program was jointly administered by the CUNY Office of Academic Affairs 
(CUNY Central) and the participating community colleges. CUNY Central devised the original 
program model and required that all of the CUNY community colleges operate an ASAP 
program beginning in the fall 2007 semester, with support and funding from the New York City 
Center for Economic Opportunity (CEO). CUNY Central was responsible for general program 
administration and any CUNY policy matters, and oversaw any modifications to the model and 
the colleges’ implementation of the program. CUNY Central was also responsible for securing 
and coordinating the MetroCard and textbook benefits for the colleges, creating marketing 
materials, conducting citywide outreach to potential students, providing professional develop- 
ment for ASAP staff, disseminating infomiation about ASAP and its effects on students, and 
managing media relations. 

The ASAP team at CUNY Central was led by the University Executive Director, who 
reported to the Senior University Dean for Academic Affairs. In addition to overseeing ASAP’s 
implementation at the colleges, the director was responsible for fiscal oversight and any 
necessary fundraising. She was also responsible for all interactions and negotiations with CEO, 
and communications with other funders, partners, and stakeholders. The director regularly 
represented ASAP in external venues, such as conferences and briefings. Finally, the director 
was responsible for CUNY’s internal evaluation of the program (discussed later). 

When the study started, the CUNY Central team also included two program coordina- 
tors, each of whom worked closely with three colleges on their program implementation. The 
program coordinators monitored program operations and served as a resource for the college 
ASAP directors. In 201 1, one of the program coordinators became the ASAP Deputy Director 
and began working closely with the University Executive Director to oversee the program. She 
supported the colleges in their governance and provision of the program, and in their opera- 
tional and budgetary planning. She also worked closely with the ASAP evaluation team to 
develop streamlined systems for data collection and reporting, played a key role in promoting 
increased collaboration and resource sharing across college sites, and was responsible for 
ASAP’s media, marketing, and outreach campaign aimed at recruiting students. She continued 
playing the program coordinator role for two of the six colleges with ASAP programs, and the 
second program coordinator oversaw the other four. The CUNY Central ASAP team also 
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included the ASAP Assistant Director of Research and Evaluation, a research associate, and an 
administrative assistant. 

Each college was responsible for managing, staffing, and implementing its ASAP pro- 
gram. Each college’s ASAP program was led by a director who reported to the college’s Vice 
President for Academic Affairs and received day-to-day support from the college’s Dean of 
Academic Affairs. The college ASAP directors closely oversaw the program’s implementation, 
supervised the college’s ASAP staff, and worked with CUNY Central. Each college’s ASAP 
program had four to six advisers. Most, if not all, had experience as advisers before working for 
ASAP. All had master’s degrees, many in counseling or social work. The college ASAP staff 
also included one career and employment specialist, clerical staff members, and part-time tutors. 
The career and employment specialists all had experience in employment and career-related 
services before joining ASAP. The tutors were typically adjunct faculty or former students who 
had moved on to four-year institutions. Other than the part-time tutors at the colleges, all ASAP 
staff worked full time for the program. During select periods, such as registration, ASAP 
advisers pitched in on college-wide tasks. 

Over the course of the study period, most ASAP staff remained in their positions. A few 
advisers at the colleges left their jobs and were replaced. The ASAP director at one of the 
colleges moved to a different position at the college in early 2012 and was replaced within a few 
months. 


CUNY Central staff and the college ASAP directors met in person monthly and com- 
municated frequently between meetings. The University Executive Director said that she spoke 
with the college ASAP directors at least once a week. The program coordinators communicated 
with the ASAP directors even more frequently, and typically visited the colleges once a month. 
The staff in the CUNY Central ASAP office met weekly and communicated daily. Similarly, 
the ASAP staff at each college typically had a staff meeting weekly and communicated daily. 
The ASAP advisers from all the colleges periodically met, as did the career and employment 
specialists. All staff gathered annually for an ASAP retreat. 

The ASAP team at CUNY Central led several internal evaluation efforts of the pro- 
gram. The research was intended to assess the effects of the program on students but also to 
improve the program. As the ASAP University Executive Director said, they “regularly use data 
to inform practice.” Each college’s ASAP staff recorded infonnation about contact with 
participating students in a centralized data management system. Those data were analyzed 
monthly by CUNY Central evaluation staff. Reports were shared with the colleges and the 
information was used to help manage and modify the program. For example, if students at a 
certain college were not meeting the benchmark to meet with their adviser twice a month, the 
CUNY Central staff and the college’s ASAP director would discuss the issue and possible 
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remedies. CUNY Central also conducted periodic surveys of ASAP students and held student 
focus groups about their perceptions of and satisfaction with the program. CUNY Central has 
also been conducting an internal quasi-experimental evaluation of the effects of ASAP on 
students’ academic outcomes using a constructed comparison group. 2 Overall, during the study 
period, the ASAP program operated with a high level of monitoring and assessment, with a 
focus on ongoing improvement. 


Requirements and Messages 

As discussed in Chapter 1, ASAP aims to accelerate students’ progress through college. ASAP 
students are required to attend college full time. They are also encouraged to take courses 
during winter and summer intersessions, complete their developmental courses early, and 
graduate within three years. This section describes how the requirements and messages were 
communicated and received and highlights the key differences between ASAP and the study’s 
control environment. 

Enroll Full Time and Take Intersession Courses 

ASAP: The program requires students to attend college full time in each fall and spring 
semester they are in the program. (Students who need fewer than 12 credits to graduate do not 
need to enroll full time in their final semester.) ASAP advisers reported that, for the most part, 
this requirement was not an issue for participating students. When students were unable to 
attend full time because of work, family, health, or other reasons, they could receive ASAP 
advising and other student services, but not the financial benefits. (These semesters counted 
toward a student’s three-year limit on ASAP services.) 

At two of the three colleges in the evaluation, the fall and spring semesters are each 
composed of two sessions: a 12-week session followed by a 6-week session. Courses taken 
during the 6-week session count toward full-time enrollment. For example, a student can take 9 
credits during the 12-week session and 3 credits during the 6-week session for a total of 12 
credits and be considered a full-time student. ASAP advisers at these colleges said that they 
encouraged students to take courses in the six-week intersessions, in some cases to achieve full- 
time status for the semester and in other cases to go beyond the 12-credit threshold and accumu- 
late more credits. ASAP advisers at the third college reported that they encouraged students to 
take courses during the two six-week summer sessions, but tended to encourage only high- 
perfonning students to take courses during the more intensive three-week winter session. 


2 To date, CUNY has published two public reports from their evaluation: Linderman and Kolenovic (2009) 
and Linderman and Kolenovic (2012). 
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Advisers discussed both full-time enrollment and enrolling in intersessions as a way to help 
accelerate students’ progress through college. For an example of how one student’s enrollment 
in intersessions helped him progress academically, see Box 3.1. 

Service contrast: Based on MDRC field research, the academic advisers who work 
with the general college population at the three colleges in the evaluation did not encourage full- 
time enrollment or enrollment in intersessions to the degree that the ASAP advisers did. One 
academic adviser who worked with the regular college population said that it was “almost 
impossible” for most students at the college to carry a full-time load. 

Take Developmental Courses Early 

ASAP: In MDRC field research interviews, ASAP advisers reported that they typically 
encouraged students to take their developmental courses during their first or second semesters 
in the program. They said that they discussed this primarily when advising students about 
course planning and registration. Developmental courses are often part of the blocked or linked 
classes that ASAP offers to participating students during their first two semesters (see the later 
section on course enrollment). 

Service contrast: In interviews, non-ASAP advisers said that they often encouraged 
students to take developmental education courses early, although not as strongly as ASAP 
advisers did. 3 A few non-ASAP advisers talked about students sometimes not heeding the 
advice to take developmental courses early, preferring to put them off. With the exception of 
first semester learning communities at one college (discussed later), non-ASAP students did not 
take blocked or linked classes with developmental courses. 

Responses to MDRC’s student survey provide evidence that the messages about devel- 
opmental courses for ASAP students and non-ASAP students were different. As shown in Table 
3.2, 47 percent of program group members reported that they “often” or “very often” heard 
faculty or staff speak about the importance of taking developmental courses in the first two 
semesters of college, compared with 31 percent of control group members. The difference of 16 
percentage points is statistically significant. (See Box 3.2 for a description of how to read the 
tables in this chapter that present findings from the MDRC student survey.) This difference is 
meaningful, although perhaps not as large as might be expected. 


’Beginning in fall 2013, CUNY instituted a policy that students with developmental education needs 
should take developmental courses immediately upon enrolling. This change occurred after the follow-up 
period for this report and had no effect on the sample members. 
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Box 3.1 


How Advising and Intersessions Propelled Frank Forward 

In his mid-30s and married with children, Frank decided to enroll in night classes at the nearby 
community college with the hope of eventually transferring to a four-year university. He had 
worked in public service for more than 10 years and knew that a bachelor’s degree would be 
required for him to move into management. Fra nk worked full time during the day and took a 
full-time course load in the evenings. 

Frank was a strong English student but struggled in math, testing into the lowest level of devel- 
opmental math courses. Knowing that math would be a stumbling block for him, he decided to 
take his first developmental math course in the summer before enrolling in ASAP, hoping he 
could remember some of the math he had learned in high school and ease himself back into 
academic life. 

Frank developed a good relationship with his adviser, who understood the unique struggles of 
nontraditional students and students attending evening courses. Frank would frequently drop by 
after his classes in the evening to say hello and ask questions. He talked with his adviser about 
topics such as course sequencing, how to interact with professors, finding scholarships, and 
planning how to complete his associate’s degree and transfer to a four-year school. Because he 
saw his career advancement as contingent on earning a degree, he wanted to stay on track. 
Frank’s adviser reported that it was comprehensive advising, along with the free textbooks, that 
proved to be most helpful to Fra nk ’s success in ASAP. 

Taking his adviser’s advice, Frank enrolled in intersessions every semester and quickly accu- 
mulated credits. He repeatedly made the Dean’s List, something he had not anticipated upon 
enrolling in school. Frank was able to graduate with an associate’s degree in just two years. 
After taking a break of one semester, he enrolled in a four-year school within CUNY that is 
highly acclaimed for his major. 


Graduate Quickly 

ASAP: ASAP’s goal is for students to graduate with an associate’s degree within three 
years. MDRC field research found that ASAP delivered the message that students should and 
can graduate within three years through written materials, program orientation, and early 
discussions with advisers, beginning immediately when students entered the program. ASAP 
advisers reported that they continued to communicate that message to students throughout their 
time in the program, both directly in conversations and indirectly through helping students map 
out the courses they needed to take in order to graduate by a certain semester. Some ASAP 
advisers at one of the colleges said they encouraged many students to graduate in two years by 
taking advantage of the winter and summer intersessions. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 3.2 

Students' First-Year Experiences: Messages Received 


Three-Year Impacts Report 


Outcome (%) 

Sample 

Size 

Program 

Group 

Control 

Group 

Difference P-Value 

Often or very often heard college faculty/staff 
speak about: 

Taking developmental courses in the first 
few semesters 

724 

47.2 

30.8 

16.4 *** 0.0000 

The goal of obtaining an associate's degree 
within 3 years 

734 

69.9 

31.8 

38.1 *** 0.0000 

Survey sample size 

742 

384 

358 



SOURCE: MDRC calculations from the MDRC student survey. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and cohort. 


Service contrast: Interviews with non-ASAP advisers indicated that while they pushed 
students to think through plans for timely graduation, they did not push graduating within three 
years to the same extent that ASAP advisers did. In field research interviews, non-ASAP 
advisers were more likely to discuss various challenges of graduating quickly than were ASAP 
advisers. They were also more likely to talk about the pace of moving through college as an 
individual issue, grounded in a student’s specific situation. 

Table 3.2 shows the proportion of sample members who reported on the MDRC student 
survey that they often or very often heard faculty or staff talk about the goal of obtaining an 
associate’s degree within three years. Far more program group members than control group 
members reported hearing that message: 70 percent compared with 32 percent. 


Student Services 

ASAP’s requirements and messages are complemented by a set of wraparound services meant 
to meet students’ academic and personal needs. Participation in these services is mandatory, 
except where noted. The program links participation in services to MetroCard receipt. CUNY 
created a data tracking system for ASAP staff members to keep track of how students are 
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Box 3.2 

How to Understand the Tables in This Report 

Many tables in this report use the format illustrated in the abbreviated table below, which 
displays some hypothetical survey and transcript data for the program and control groups. The 
first row shows that 82.6 percent of program group students and 73.4 percent of control group 
students ever spoke with an adviser. The average outcomes of control group members serve as a 
proxy for what would have happened to program group members had they not been offered the 
program. 

The difference column in the table shows the difference in the average outcomes of the two 
research groups. The difference is the estimated effect of the program. For services such as 
advising, this number represents the average increase (or decrease) in services experienced as a 
result of the program. For target outcomes, such as credits earned, this number represents the 
estimated value added of being assigned to the program, above and beyond what students would 
have achieved if assigned to the control group. In the table below, the estimated average effect 
of the program on credits earned can be calculated by subtracting 7.9 from 8.7, yielding an 
estimated effect of 0.8. This effect indicates that, as a result of being offered the program, 
students earned an estimated additional 0.8 credits compared with what they would have earned 
had they not been offered the program. 

This difference represents the estimated effect rather than the true effect, which is impossible to 
know because we cannot know exactly what would have happened to program group students 
had they not been offered ASAP. This counterfactual world is approximated by the control 
group. Had a different sample of students participated in the study or if the same group of 
students had been randomized in a different way, a different estimate would have resulted. 

Estimated effects marked with one asterisk or more are, by convention, considered statistically 
significant. The number of asterisks corresponds to the p-value, which indicates the likelihood 
that an effect at least as large as the one observed in the study would have occurred if the true 
effect were zero. One asterisk corresponds to a 10 percent probability; two asterisks, a 5 percent 
probability; and three asterisks, a 1 percent probability. The second estimated effect in the table 
excerpt has two asterisks, indicating that the estimated effect is statistically significant at the 5 
percent level; that is, there is less than a 5 percent chance of observing an estimated effect this 
large or larger if the opportunity to participate in the program had no true average effect on 
credits earned. In other words, there is a 95 percent level of confidence that the opportunity to 
participate in the program had a positive effect on the average number of credits earned. 


Outcome 

Program 

Group 

Control 

Group 

Difference 

P-value 

Ever spoke with an adviser (%) 

82.6 

73.4 

g 2 *** 

0.0032 

Credits earned 

8.7 

7.9 

0.8 ** 

0.0427 
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participating in the program: contacts with advisement and career services are logged monthly 
in the ASAP database, while tutoring visits and attendance in the ASAP seminar are tracked 
weekly by college staff and reported to CUNY Central each semester. 

Comprehensive Advisement 

ASAP: Acknowledging the variety of barriers to academic success students may face, 
ASAP provides students with comprehensive advising not only on academic issues but also on 
social and interpersonal issues. During the course of the study, ASAP students were required to 
meet with their assigned adviser in person twice per month throughout each semester; advisers 
also sometimes communicated with students by phone, e-mail, or text message. Advising 
appointments were tracked, and attendance was linked to students’ receipt of monthly Metro- 
Cards. ASAP students were assigned an adviser during their first semester and usually contin- 
ued to see the same person throughout their college careers. 

ASAP advisers serve only ASAP students. During the study, they typically held case- 
loads of 60 to 80 students each semester. This caseload was substantially smaller than the 
national median in community colleges, which the National Academic Advising Association 
puts at 441 students per adviser, 4 and the average at each school where ASAP operates. At the 
three colleges in the study, the ratios ranged from 600 to 1,500 students per adviser. 5 The small 
caseloads allowed advisers to meet with students more frequently and for longer durations. 

ASAP advisers were trained to field a wide variety of topics. In addition to covering the 
gamut of academic topics, including getting acclimated to college, choosing classes, and 
picking a major, ASAP advisers helped students with soft skills, such as study habits and time 
management, how best to balance home, work, and school demands, and extracurricular 
activities and campus life. Indeed, among survey respondents who saw an adviser, program 
group students reported discussing an average of eight topics with advisers, compared with five 
topics for control group students. (See Table 3.3) ASAP advisers kept their schedules open to 
students for as much time as possible during the day; at schools where ASAP included night 
and weekend students, ASAP advisers dedicated to night and weekend students kept hours late 
into the evening and for several hours on Saturdays and Sundays. 

Advisers sometimes interacted directly with students’ professors or financial aid offic- 
ers to resolve issues. ASAP advisers reported that they took a forward-looking approach to 
advising, encouraging students to think early on about aligning their college experience with 
their career goals and planning to transfer to four-year institutions. ASAP advisers also helped 


4 Robbins (2013). 

5 Linderman et at. (2011). 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 3.3 

Students' First-Year Experiences: Advising 
Three-Year Impacts Report 



Sample 

Program 

Control 


Outcome 

Size 

Group 

Group 

Difference P-Value 

Ever spoke with an adviser (%) 

739 

95.0 

80.4 

14.6 *** 0.0000 

Average number of times spoke with an 
adviser in first year 





First semester 

718 

21.1 

3.7 

17.4 *** 0.0000 

Second semester 

718 

16.6 

2.0 

14.6 *** 0.0000 

Among those who spoke with an adviser: 





Average time spent during visit to adviser (%) 




15 minutes or fewer 

647 

24.5 

42.2 


16-30 minutes 

647 

63.6 

47.4 


31 minutes or more 

647 

12.0 

10.4 


Average number of topics discussed with 
adviser 

648 

7.5 

5.1 


Topics discussed with adviser (%) 





Academic goals 

648 

91.1 

64.5 


Academic progress 

648 

94.7 

61.7 


Course selection 

648 

92.5 

89.2 


Major 

648 

75.1 

72.4 


Requirements for graduation 

648 

79.0 

60.6 


Internships 

648 

39.4 

11.4 


Job opportunities 

648 

45.5 

15.6 


Career planning 

648 

45.1 

24.1 


Transfer credit policies, probation, 

648 

60.6 

44.4 


and drop/add policies 
College services such as financial aid, 

648 

73.2 

43.5 


tutoring, and counseling 
Personal matters 

648 

48.7 

17.1 


Something else 

648 

8.9 

3.1 


Quality of advising scale(%) a 





Low 

647 

4.4 

28.3 


High 

647 

37.5 

10.5 


Survey sample size 

742 

384 

358 



(continued) 
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Table 3.3 (continued) 

SOURCE: MDRC calculations from the MDRC student survey. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and cohort. 

Italic type indicates nonexperimental data. Significance tests are not calculated for nonexperimental 
data. 

a The scale is based on responses to five questions about the quality of advising received. "Low" is 
the percentage of sample members whose ratings of adviser quality were one standard deviation below 
the mean, indicating a comparatively lower rating; "high" is the percentage of sample members whose 
ratings of adviser quality were one standard deviation above the mean, indicating a comparatively higher 
rating. The questions are listed in Appendix B. 


students with personal issues as they arose, either within the advising context or by refer- 
ring students to other resources. For an example of how one student benefited from advis- 
ing, see Box 3.3. 

Service contrast: While dedicated, the colleges’ non-ASAP advisers managed very 
large caseloads that typically did not permit the more personalized touch that students experi- 
enced in ASAP advising. Students who were not in ASAP were not required to go to advising 
except at registration time; once students had passed all their developmental requirements, some 
colleges allowed them to register online without first seeing an adviser. While students were 
able to seek out advising as often as they wished, there was no external incentive to do so. 
Additionally, most advisers focused on academic topics, such as choosing majors and classes; 
students with personal issues were referred to counselors. 

Responses to MDRC’s student survey provided additional infonnation on how the ad- 
vising component of ASAP was experienced by students. (See Table 3.3.) Program group 
survey respondents reported seeing their advisers more often than control group students during 
the first year. Program group students’ average number of visits was 21 in the first semester and 
17 in the second semester, a dramatic increase over control group survey respondents’ average 
of 4 in the first semester and 2 in the second semester. Among those who saw an adviser, 
program group students were more likely than control group students to rate advising as high 
quality during their first year. See Appendix B for the list of questions students were asked 
about their advising experience. 

Program group survey respondents also reported spending more time with advisers dur- 
ing each visit. Of students who saw an adviser, 64 percent reported spending 16 to 30 minutes 
on average with an adviser per session, compared with 47 percent of control group survey 
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Box 3.3 


Michelle Takes Advantage of Comprehensive Advising 

The daughter of a single mother, Michelle grew up in a low-income neighborhood in one of 
New York City’s outer boroughs. She enrolled in community college immediately after gradu- 
ating high school. She already had a college trajectory in mind, with plans to transfer to a four- 
year university after taking courses at the community college. 

Michelle quickly formed a personal bond with her ASAP adviser, and they partnered to make 
sure she could graduate even more quickly than the three-year ASAP deadline. Michelle 
attended advising more frequently than required to make sure she was on track both in the 
ASAP program and in terms of the requirements for transferring to her preferred four-year 
university. She also e-mailed and texted her adviser with brief or time-sensitive questions. 
Together, they created an ambitious and personalized plan for Michelle’s needs, leading her to 
accumulate all the credits she needed for her associate’s degree in just three semesters, includ- 
ing intersessions. Michelle appreciated the program’s comprehensive advising, which covered 
topics from academics and extracurricular activities to time management and personal issues. 

Michelle arrived in college needing one developmental course, a math course that she initially 
failed but was able to pass by retaking it alone during an intersession. She credited the ASAP 
tutoring with her success in this course. Michelle also found the financial supports of the 
program incredibly helpful, relieving her financial stress both about books, which she could not 
otherwise afford, and transportation. Before ASAP, Michelle worried about transportation costs 
preventing her from attending class every day. 

By the end of her third semester, Michelle transferred to a four-year CUNY college that is 
highly acclaimed for her major. There, she has earned all As and Bs. Based on her credit 
accumulation, she is on track to receive her bachelor’s degree within four years of her 
original enrollment in ASAP. 


respondents. Control group survey respondents who saw an adviser were much more likely to 
report sessions lasting an average of 15 minutes or less, 42 percent compared with 25 percent of 
program group survey respondents. While respondents in both groups reported talking with 
advisers about topics such as course selection and major, program group students reported 
covering more topics with their advisers, including academic goals, academic progress, re- 
quirements for graduation, and career planning. For instance, 95 percent of program group 
survey respondents who saw an adviser reported discussing academic progress, compared with 
just 62 percent of control group survey respondents. Nearly half of program group survey 
respondents who saw an adviser reported discussing career planning, compared with just one- 
fourth of control group survey respondents. Program group survey respondents who saw an 
adviser were also much more likely to report discussing personal matters than control group 
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survey respondents, 49 percent compared with 17 percent. While these outcomes are all non- 
experimental and tests of statistical significance were not carried out on them, these gaps are 
large enough to imply that ASAP students’ advising experiences had more depth and breadth 
than those of students not in ASAP. 

The MDRC student survey asked only about first-year college experiences. However, 
CUNY ASAP tracking data indicate that a relatively high number of advising visits continued 
in the second and third years for ASAP students, with advisers reporting that most students 
visited six or more times each semester in the third, fourth, and fifth semesters. At one school, 
most students visited eight times or more each semester. While not as many as the number of 
visits students reported in the first year, this number does indicate that ASAP students continued 
to see their advisers at high rates throughout the three years of the program. 6 

Several ASAP staff said that they felt advising was the most important nonfinancial 
component of the program for students. Indeed, numerous students who responded to the 
MDRC student survey with additional feedback commented that their advisers were important 
to their academic success. Furthermore, ASAP students who participated in CUNY’s surveys 
and focus groups about the program said that advisement was one of the most helpful program 
components. 7 

Career Services 

ASAP: In addition to semimonthly advising visits, ASAP students were required to 
meet with an ASAP-dedicated career and employment specialist (CES) once per semester. This 
requirement aimed to get students thinking about jobs and career planning early. The CES 
covered a number of topics with students, including finding part-time work, balancing work and 
school, finding scholarships, writing resumes, networking, and aligning college majors with 
career paths. The CES also hosted career fairs and job talks that ASAP students could attend, 
and in some semesters organized visits to job sites so students could see first-hand how a 
particular employer or field operated. 

At two of the schools during the study, the CES focused primarily on long-term goals, 
such as giving students career inventories, helping students connect majors and careers, and 
finding career-relevant internships and scholarships. The CES encouraged students to plan 
ahead for their careers and prioritize education. At the third school, the CES was much more 


6 The sample of students represented in the CUNY ASAP tracking data used to estimate trends for advising 
and career and employment specialist (CES) visits in the second and third years is slightly different from the 
study’s program group. It included a few ASAP students who were not in the program group and did not 
include program group students who did not enroll in ASAP. 

7 Linderman and Kolenovic (2012). 
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focused on short-term goals, helping students find part-time work while they were in school and 
introducing them to local work opportunities. At all three schools, advisers reported occasional- 
ly conferring with the CES about how to help students apply for scholarships or achieve their 
career goals. 

Service contrast: Non-ASAP students had access to the colleges’ career services of- 
fices. There was no external incentive to visit these offices, although staff there reported being 
quite busy, especially with students who were in their last semester of school. These staff 
helped students with most of the same topics as the ASAP CES, but typically did so on a one- 
time basis. 

Student survey responses indicated that the majority of program group students experi- 
enced ASAP’s CES component. Eighty percent of program group survey respondents reported 
that they had spoken with staff in career or employment services in their first year, compared 
with 29 percent of control group students. (See Table 3.4.) Program group students’ average 
number of visits in the first semester was 4.9, compared with control group students’ average of 
1 .0. In the second semester, program group students reported meeting with the CES an average 
of 4.1 times, compared with control group students’ average of 0.6 times. 

As was noted about advising, MDRC’s student survey asked only about first-year col- 
lege experiences. However, CUNY ASAP tracking data indicated that ASAP students continued 
to visit the CES at comparatively high rates in the second and third years, with most students 
visiting the CES two to three times each semester in the third, fourth, and fifth semesters. The 
majority of ASAP students continued to visit the CES in the third semester. The percentage of 
ASAP students visiting the CES began decreasing in the fourth and fifth semesters. 

Tutoring 

ASAP: ASAP students were required to attend tutoring during their developmental 
courses and any semesters they were on academic probation — a grade point average (GPA) of 
2.0 or lower. Each school hired tutors dedicated to ASAP students, often housed in a centrally 
located space separate from the regular tutoring center on campus. Students had to attend one 
hour of tutoring per week for each developmental course in which they were enrolled. If 
students had previously failed the course and were retaking it, they had to attend two hours of 
tutoring per week for the course. Students brought in forms for the tutors to sign confirming 
their attendance, which were checked by ASAP administrative staff before distributing Metro- 
Cards each month. ASAP students could spend more time with ASAP tutors than required, but 
that time above the requirement was not tracked. Advisers reported encouraging students 
struggling with their college-level courses to attend tutoring, although it was not required. They 
said, however, that only some students did so. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 3.4 

Students' First-Year Experiences: Career Services 


Three-Year Impacts Report 


Outcome 

Sample 

Size 

Program 

Group 

Control 

Group 

Difference 

P-Value 

Ever met with career or employment services 
staff (%) 

736 

79.5 

28.8 

50.6 *** 

0.0000 

Average number of times spoke with career or 
employment services staff 
First semester 

719 

4.9 

1.0 

2 g *** 

0.0000 

Second semester 

721 

4.1 

0.6 

3 5 *** 

0.0000 

Survey sample size 

742 

384 

358 




SOURCE: MDRC calculations from the MDRC student survey. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and cohort. 


ASAP tutors said that because they met only with ASAP students, they were able to 
build trust with students and understand the material covered on a deeper level. Although ASAP 
tutors were hired for only English and math, they often helped students with other subjects as 
well, such as writing history papers or doing physics problem sets. As noted above, the tutors 
were typically adjunct faculty or former students now at four-year institutions. Some tutors also 
visited classes with high concentrations of ASAP students to provide supplemental instruction 
and tutoring. 

Service contrast: Non-ASAP students had access to the colleges’ tutoring centers, 
which offered tutoring in all academic subjects, although some subjects, such as foreign 
languages, were housed in different locations. Visits to tutoring centers were voluntary unless 
students were either on academic probation, in which case they had to attend in accordance with 
their academic plan, or referred by a professor, which advisers reported was quite rare. 

The findings from MDRC’s student survey indicated that program group students were 
more likely to receive tutoring than were control group students. Seventy-four percent of 
program group survey respondents reported receiving tutoring outside of class, compared with 


32 







only 39 percent of control group survey respondents, a statistically significant difference. (See 
Table 3.5.) Program group students were also 26 percentage points more likely to receive in- 
class tutoring. On MDRC’s student survey, program group students reported an average number 
of tutoring visits of 13 in the first semester and 1 1 in the second semester. This number corre- 
lates with the tutoring requirement for developmental classes; as students passed their develop- 
mental courses and were no longer required to attend tutoring, they were less likely to attend. In 
both semesters, however, there was a significant difference between program and control group 
students in tenns of average number of tutoring sessions. Note that the survey question asked 
about tutoring broadly, not only about ASAP tutoring, so students may have reported using the 
college’s general tutoring program in addition to, or instead of, ASAP tutoring. 


Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 3.5 

Students' First-Year Experiences: Tutoring 
Three-Year Impacts Report 

Sample Program Control 


Outcome 

Size 

Group 

Group 

Difference 


P-Value 

Ever received in-class tutoring from someone 
other than the instructor (%) 

738 

49.4 

23.8 

25.7 

*** 

0.0000 

Ever received tutoring outside of class (%) 

736 

73.6 

39.4 

34.2 

*** 

0.0000 

Average number of times used tutoring services 
outside of class 







First semester 

724 

13.0 

4.0 

9.0 

*** 

0.0000 

Second semester 

730 

10.6 

2.5 

8.1 

*** 

0.0000 

Survey sample size 

742 

384 

358 





SOURCE: MDRC calculations from the MDRC student survey. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and cohort. 
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Other Student Services 


ASAP offered students a few services that were voluntary. One such service was the 
social work intern (SWI), a CUNY Hunter College master’s degree candidate who worked in- 
house three days per week at each campus. ASAP students could see the SWI one on one as 
needed for in-depth counseling and social work services. This component of the program 
received mixed responses at the schools. Staff and students thought an on-site social worker 
could have been valuable, but found that changing the SWI each semester was neither popular 
with nor effective for students. By the last year of ASAP for the study’s second cohort, two of 
the colleges had eliminated the position. The third had converted the component into group 
sessions, which focused on issues that concern underserved populations such as first-generation 
college students and offered participants a safe setting in which to talk about these issues. Due 
to the sensitive nature of this component, data were not collected about ASAP students’ visits 
with the SWI. 

In addition, each semester, a handful of ASAP students across all of CUNY’s campuses 
were selected to participate in the ASAP Student Leadership Program. These students partici- 
pated in workshops aimed at developing leadership and public speaking skills, including 
sessions on teamwork, diversity, and advocacy. They would then complete a team project, such 
as hosting a domestic violence awareness session in the community or creating a mentoring 
program for low-income high school students. Because so few students participated in this 
component, data were not collected about it. 


Course Enrollment 

ASAP did not make changes to pedagogy, curricula, or anything else that happened inside of 
the classroom. The ASAP students in this study experienced the same faculty and curriculum 
as the other students at the colleges. The ASAP program did, however, make two changes to 
students’ course enrollment patterns. ASAP students participated in an ASAP seminar course, 
similar to a student success course, in their first few semesters in the program. ASAP students 
were also able to enroll in blocked or linked courses with other ASAP students in their first 
two semesters. 

In addition, students were able to register for classes early in every semester they partic- 
ipated in the program. This feature allowed ASAP students to create convenient schedules and 
have a better chance of enrolling in all the classes they need. Early registration may be especial- 
ly beneficial for students who need to enroll in classes that are often oversubscribed, such as 
popular general education requirements or developmental courses, and for students in their final 
semesters as they complete the last courses they need to graduate. Many students responding to 
the MDRC student survey noted early registration as a valuable benefit of the program. No data 
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are available on the proportion of ASAP students who registered early, although advisers 
reported that the majority did. 

ASAP Seminar 

ASAP: ASAP students attended an ASAP seminar each semester for their first three to 
four semesters in ASAP. This was a noncredit course, substituting for a student success course, 
and covered infonnation about the school, such as available student services and campus 
navigation, soft skills, including study and problem-solving skills, and stress management. The 
seminar included early planning for transfer to four-year institutions and career planning. The 
class also brought in occasional guest speakers to discuss topics such as leadership and 
community involvement. Depending on the semester, advisers, the CES, or facilitators taught 
the seminar, although the curriculum was similar across schools. The course was meant to 
foster a sense of community among ASAP students and educate them about campus resources 
and soft skills. 

At two of the schools, the ASAP seminar in the first semester was replaced with the 
school’s student success course, open only to ASAP students. The curriculum was considered to 
be similar to the ASAP seminar curriculum. 

Service contrast: Non-ASAP students at the school offering all students first-semester 
learning communities also took a mandatory student success course in the first semester, 
although they did not take any courses analogous to the ASAP seminar in subsequent semesters. 
At the school where first semester learning communities were available to students who were at 
college level in English, students took a student success course in their learning community. 
The course, however, was otherwise optional to non-ASAP students. At the third school, 
students had the option of enrolling in a student success course in their first year to become 
acquainted with campus resources and to leam good study skills. This course was not linked 
with other courses, and some students chose not to enroll. 

Data were not available for ASAP seminar participation since the class was not consist- 
ently recorded in course catalogs and databases across campuses and semesters. Qualitatively, 
advisers and CUNY Central administrators reported that the vast majority of students took the 
ASAP seminar for at least one semester, although a few students did not take the course in later 
semesters, especially if it conflicted with courses required for their majors. 

Blocked and Linked Courses 

ASAP: All three schools offered some form of blocked or linked courses for students in 
their first year. Blocked courses refer to courses scheduled back to back, while linked courses 
refer to courses offered as a set but not always scheduled back to back. All schools were 
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required to work with academic departments to reserve seats or sections specifically for ASAP 
students. These seats and sections remained reserved for ASAP students until the end of 
registration. The main purposes of the blocked and linked classes were to ensure that ASAP 
students could take some of their classes with other ASAP students, fostering a sense of 
community, and to give students convenient, consolidated schedules so they could make the 
most of their time on campus. Unlike classical learning communities, these courses did not 
include integrated curricula or interaction among faculty. However, they did give students the 
chance to have a community-like experience by sharing courses with other ASAP students. 

During the study, each school created a blocked or linked class arrangement that typi- 
cally included a core course, such as English, blocked or linked with one or more other courses 
applicable to a great number of students, such as a speech course and the ASAP seminar. 
Blocked and linked courses were created with both developmental and college-level courses at 
the core, so that students at either level could participate. The other courses were all college- 
level courses. Advisers consistently encouraged ASAP students to take blocked or linked 
courses, although enrollment was not required. Additionally, students were not required to take 
all courses in a block; if they were only able to fit one course into their schedule, they were still 
encouraged to do so. 

The implementation of blocked and linked courses varied by campus. At one school, 
students were offered blocked classes in their first two semesters, typically a speech, social 
science, or history class (a college-level course applicable to all majors) blocked with either 
college-level English or math (since the program’s launch in 2007) or remedial English or math 
(starting in spring 2011). Advisers encouraged students to take both classes in a block, but 
enrollment depended on how the courses fit into students’ schedules. When students could not 
attend both courses in the block, advisers suggested taking at least one of the courses. The 
advisers stressed the benefits, which included being in a course with only ASAP students and 
taking a course that is slightly smaller than the average class size. In addition, ASAP advisers 
kept in contact with the professors of blocked courses to make sure students were staying on 
track. This connection provided students with additional support and gave advisers an early 
warning system if their students began struggling academically. 

At the second school, most ASAP students enrolled in preexisting learning communi- 
ties in the first semester. These learning communities consisted of three or four courses, typical- 
ly built around a developmental or college-level English course, and included a student success 
course, replacing the ASAP seminar, in which only ASAP students enrolled. The curricula of 
these courses were often linked, with professors collaborating and reviewing students’ progress. 
In the second semester, students were able to enroll in an ASAP-designated English course 
blocked with the ASAP seminar, which were linked only in terms of scheduling, not curricula. 
These classes appeared first on students’ schedules: they picked the English and seminar link 
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that they wanted, then selected all of their other classes around them. There were more English 
classes than seminars, so multiple English classes were linked with each seminar. However, 
only ASAP students enrolled in the courses. 

The third school offered blocked courses during students’ first two semesters. The 
blocks consisted of specific sets of three to four courses, scheduled back to back with the 
majority of seats reserved for ASAP students. Only after ASAP students had registered were 
non-ASAP students permitted to register for the remaining seats in the courses. The blocks 
included courses such as English 101, Introduction to Business, Communications, and devel- 
opmental math courses, all of which are applicable to a wide variety of majors. The curricula of 
the courses were not linked, and students could take as many or as few as they preferred. 
Advisers encouraged students to take as many of these courses as they could fit in their sched- 
ules, as the courses were made up of entirely or mostly ASAP students and scheduled in 
convenient blocks. 

Analysis of course enrollment data indicated that participation in blocked and linked 
courses varied. Across the three schools, about one-third of students enrolled in all of the 
components in a block. However, many more students enrolled in one course in a block, just as 
advisers suggested for students unable to take the full set. Perhaps most important, a majority of 
program group students were enrolled in one or more classes with a concentration of ASAP 
students. S * * 8 The percentage of ASAP students enrolled in at least one of these courses ranged 
from more than 60 percent at one school to nearly 85 percent at another. This was largely, 
although not exclusively, accomplished by the blocked or linked course offerings. 

After the first year, blocked or linked courses are no longer offered to ASAP students. 
Advisers reported that some groups of students who became friends in the first year planned 
ahead to take courses together. Students were often advised to try to consolidate their courses as 
much as possible to create convenient school schedules, but whether this happened varied by 
student. No uniform consolidated schedules were offered to students after the first year. 

Given that, across the three schools, enrollment rates in all courses in a block were 
somewhat low, it is unlikely that these courses as designed had a substantial impact on ASAP 
students’ success. However, the much higher enrollment rates in courses with a concentration of 
ASAP students indicates that students did experience — and may have benefited from — social 
interaction with ASAP peers. Students also likely benefited from being able to access reserved 
seats in courses at convenient times or in oversubscribed courses. 


S A class with a concentration of ASAP students is defined as any class in which five or more ASAP stu- 

dents were enrolled. These five or more students included ASAP students who were not part of the MDRC 

study and excluded students in the study randomly assigned to the program group but who were not participat- 

ing in ASAP. 
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Service contrast: In general, non- ASAP students were not offered blocked or linked 
courses such as those offered to ASAP students. At the school that offers learning communities 
to its ASAP students in the first semester, non-ASAP students are also offered learning com- 
munities in the first semester, and a majority of non-ASAP students enrolled. At another school, 
first-semester learning communities were available to non-ASAP students who were reading on 
a college level, although enrollment rates were lower than at the school offering learning 
communities to all new students. At both of these schools in the second semester and beyond, 
and at the other school in all semesters, there were no other widely available blocked or linked 
course options for non-ASAP students. 


Financial Supports 

ASAP provides a few key financial supports to participating students. This section discusses 
those supports and the differences between those supports and what the evaluation’s control 
group received. 

Tuition Waiver 

ASAP: ASAP waives the difference between a student’s tuition and fees and financial 
aid, specifically, the sum of the federal Pell Grant and the New York State Tuition Assistance 
Program grant, CUNY students’ two main need-based sources of aid . 9 In order to be eligible for 
a waiver, ASAP students must apply for financial aid by completing the Free Application for 
Federal Student Aid and be eligible for financial aid. 

During the study period, before or at the start of each semester, the college ASAP direc- 
tors and financial aid offices worked together to apply waivers as needed so that ASAP students 
did not pay any tuition or fees. The waiver was applied to accounts internally, not paid directly 
to the students, and students did not get letters or other communication about the precise amount 
of the waiver. As a result, students typically knew that the waiver existed but did not know how 
much it amounted to. 

At the two colleges where the fall and spring semesters are split into a longer main ses- 
sion and a shorter intersession, the ASAP tuition waiver covered all sessions. At the third 
college, the tuition waiver covered the fall and spring semesters. Some additional funds were 


9 The term “unmet need” is used in higher education to refer to the difference between the total cost of 
attending college and financial aid. The total cost of attendance includes tuition and fees as well as other 
education-related expenses, such as textbooks, transportation, and room and board (The Institute for College 
Access and Success, 2009). ASAP’s tuition waiver covers only tuition and fees. As discussed later, ASAP 
provides monthly MetroCards and free use of textbooks regardless of a student’s amount of financial aid (or 
level of unmet need). 
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available for ASAP advisers at that college to allocate at their discretion to cover classes in 
winter and su mm er. Typically, advisers allocated those funds for students with a 3.0 GPA or 
higher. 


Recall that one of the eligibility criteria for the evaluation was having family income 
below 200 percent of the federal poverty level or being eligible for a federal Pell Grant. CUNY 
analysis of financial aid and ASAP tuition waiver data showed that the vast majority of program 
group members received enough financial aid to cover their tuition and fees and thus did not 
need the ASAP tuition waiver. As Table 3.6 shows, only 9 percent of program group students 
received an ASAP tuition waiver during their first semester in the study. The proportion of 
students who received a waiver remained at about that level during the first two years of the 
study. During the third year, between 3 and 5 percent received a waiver. (As is discussed in 
Chapter 4, the proportion of sample members enrolled in college declined over time.) 


Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 3.6 

Three- Year Tuition Waiver Receipt Among ASAP Students 


Three- Year Impacts Report 


Outcome 

Program 

Group 

Received tuition waiver (%) 


First semester 

8.9 

Second semester 

9.3 

Third semester 

10.4 

Fourth semester 

10.6 

Fifth semester 

4.9 

Sixth semester 

3.3 

Average waiver received ($) 

57 

Average waiver amount per semester among those receiving a waiver ($) 

719 

Sample size 

451 


SOURCE: MDRC calculations from tuition waiver analysis performed by CUNY ASAP office. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

Outcomes shown in italic type are calculated for a proportion of the full sample. 
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The average tuition waiver among program group members who received a waiver was 
$719 per semester, as shown in Table 3.6. The average total tuition and fees at the three colleges 
in the study for full-time enrollment during the 2010-2011 academic year was $1,777 per 
semester. 

Service contrast: For the most part, students outside of ASAP did not receive any simi- 
lar benefit. Students in a program called College Discovery were eligible for a tuition waiver, 
but very few students in the study’s control group were in that program. 10 

Since a relatively small proportion of program group members received a tuition waiver, 
that component of ASAP likely contributed less to the overall service contrast than some of the 
other components. The availability of a tuition wavier as part of ASAP’s package of services, 
however, may have affected even students who did not receive it. Some students may have been 
more likely to enroll in college (and enroll full time) if they knew they would not need to worry 
about how they might cover their tuition and fees. (If an ASAP-like program were provided to a 
group of students whose financial aid was less likely to cover their tuition and fees in full, the 
tuition waiver might make a bigger difference; it would then also cost more.) 

Free MetroCards 

ASAP: ASAP provides students with free monthly MetroCards, contingent on partici- 
pation in key program services, for use on New York City subways and buses. When the study 
began in spring 2010, a monthly MetroCard cost $89. By December 2010, the cost had in- 
creased to $104, and by the end of the three-year follow-up period in summer 2013, the cost 
was $112. At each college, the ASAP office distributed MetroCards to students each month. 
ASAP students were eligible for MetroCards regardless of the amount of financial aid they 
received. 

As discussed above, ASAP linked students’ receipt of MetroCards with successfully 
meeting program requirements. ASAP advisers reported that they often used the MetroCard as 
an incentive to encourage compliance, reminding students that if they did not meet program 
requirements, they would not receive a MetroCard for the following month. Advisers said that 
they sometimes withheld MetroCards for noncompliance, but the vast majority of students in 
ASAP received a MetroCard in most or all of the months in which they were taking classes (not 
shown in a table). 

Service contrast: Based on MDRC’s field research, monthly MetroCards were not 
provided to non-ASAP students. Table 3.7 shows that almost all program group students (93 


10 Based on CUNY Institutional Research Database data, 4 percent of control group members participated 
in College Discovery during the follow-up period for this report. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 3.7 


Students' First-Year Experiences: Financial Supports 
Three- Year Impacts Report 


Outcome (%) 

Sample 

Size 

Program 

Group 

Control 

Group 

Difference 

P-Value 

Received financial assistance with 






transportation 

729 

93.4 

7.1 

86.3 *** 

0.0000 

Received all textbooks free of charge 

729 

76.9 

5.7 

71 2 *** 

0.0000 

Survey sample size 

742 

384 

358 




SOURCE: MDRC calculations from the MDRC student survey. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and cohort. 


percent) reported on the MDRC student survey that they had received financial assistance 
with transportation during their first year in the study. In sharp contrast, very few control 
group members (7 percent) reported that they had received any financial assistance with 
transportation. This finding indicates a substantial difference in the experiences of ASAP 
students compared with non-ASAP students. To see how one student responded to the 
MetroCard, see Box 3.4. 

Free Use of Textbooks 

ASAP: ASAP provides free use of textbooks for all courses for participating students. 
During the study period, at the start of each semester, the ASAP office provided paperwork to 
students listing their courses for the semester. The students took the paperwork to the college 
bookstore where they picked up their books. Students had to return all their textbooks by the 
end of the semester, or else they were charged for them. ASAP staff estimated that textbooks for 
a full-time course load at CUNY’s community colleges might cost about $270 a semester. 

Service contrast: Based on MDRC field research, students outside of ASAP did not 
receive free textbooks for their courses. Table 3.7 shows that more than three-fourths of the 
program group students reported receiving all of their textbooks for free during their first year 
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Box 3.4 

Javier and the Value of the MetroCard Incentive 

Javier is a Latino man who enrolled in community college immediately after high school. He 
was not interested in college, but his parents, who do not have college degrees themselves, 
convinced him it was important to continue his education. Javier took the CUNY placement 
exam and found out that he would have to take developmental courses in reading and math. 

Javier had struggled with math throughout high school and continued to struggle in college, 
failing his developmental math course twice before his ASAP adviser suggested he retake the 
course during an intersession, when he could focus on just that class. That time, he passed and 
got through the requirement. 

Javier’s ASAP adviser found that he really struggled to motivate himself, since he was not very 
interested in his chosen major or in the classes he had to take. He had followed his parents’ 
advice to enroll in college after high school, but he did not want to continue on to a bachelor’s 
degree. Javier planned to enter the workforce as soon as he got his associate’s degree. His adviser 
talked with him about the bigger picture of higher education and careers, but Javier felt com- 
pletely burned out on school. 

Although Javier was not interested in many of his classes, he was highly motivated by the 
MetroCard incentive and always made sure to meet the minimum requirements to receive it. He 
enrolled full time, attended advising as required, met with the career and employment specialist 
each semester, and attended tutoring for his developmental courses. His adviser said that these 
benchmarks, linked to receipt of the MetroCard, were the only things keeping him on track 
academically. Javier himself even admitted he could have used more motivation in school. 

Javier received his associate’s degree at the end of his third year, exactly on track with ASAP’s 
deadline. He then entered the workforce. 


in the study, compared with only a small proportion of control group students. Again, this 
result represents a substantial difference in the experiences of ASAP students and non- ASAP 
students. 

ASAP Transfer Scholarship 

Each year, a small number of ASAP students received the ASAP Transfer Scholarship, 
which provided selected students with a 3.0 GPA or better a $500 per semester scholarship for 
four semesters to attend CUNY colleges and complete their four-year degrees. Scholarship 
recipients could also receive academic advising in addition to the advising offered by the four- 
year schools. Approximately eight students received this scholarship each year. 
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Student Finances 

ASAP’s tuition waiver, transportation assistance, and free use of textbooks were in- 
tended to facilitate and support students’ participation in college, particularly full-time enroll- 
ment. As discussed above, all students whose financial aid did not cover their tuition and fees 
received a tuition waiver, and most if not all students received free MetroCards and free use of 
textbooks. Table 3.8 shows some measures from the MDRC student survey that indicate how 
sample members financed college and to what degree different financial issues affected their 
schooling and provide information about their employment. 

Students in the program group were much less likely than students in the control group 
to report that they relied on their parents, relatives, or friends to help pay for college. 1 1 Program 
group members were also less likely than control group members to say that they relied on 
student loans. These findings suggest that ASAP’s tuition waiver and possibly other financial 
supports affected how some students financed their education, reducing their need to take on 
debt and accept help from others. 

As Table 3.8 shows, one in every five control group members (21 percent) reported that 
they chose not to register for college full time because they could not afford it. Very few 
program group members (6 percent) said the same. Facilitating and supporting full-time 
enrollment is central to ASAP’s program model, and these findings suggest that the program 
made a big difference for students. (Chapter 4 discusses the program’s impact on actually 
enrolling full time.) Table 3.9 also shows that almost half of program group members (49 
percent) said they had few or no concerns about their ability to pay for college, compared with 
only about one-third (36 percent) of control group members. 

Almost one-fourth of control group students (23 percent) said that they often or very of- 
ten did not buy textbooks because of the cost. The ASAP program, with its free use of text- 
books, reduced that proportion substantially — only 1 1 percent of the program group students 
reported the same. Very few students in the control group reported missing class often or very 
often because they could not pay for transportation; the proportion of program group students 
who reported missing class for those reasons was similar. More program group students than 
control group students, however, reported that they never missed class because they could not 
pay for transportation (77 percent, compared with 61 percent; not shown in the table). 


1 'it is not clear why 8 percent of control group members said they received ASAP financial resources. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 3.8 

Students' First-Year Experiences: Student Finances 


Three-Year Impacts Report 



Sample 

Program 

Control 



Outcome 

Size 

Group 

Group 

Difference 

P- Value 

Funding sources for college (%) 






ASAP financial resources 

730 

83.0 

8.2 

74_g *** 

0.0000 

External financial aid J 

730 

90.4 

87.3 

3.2 

0.1766 

Student loans ’ 5 

730 

5.8 

9.7 

-3.8 * 

0.0528 

Parents, relatives, partners, or friends 

730 

17.0 

28.3 

4 *** 

0.0002 

Employment 

730 

19.9 

24.1 

-4.2 

0.1674 

Credit cards 

730 

5.0 

5.7 

-0.7 

0.6878 

Personal savings 

730 

15.4 

17.6 

- 2.2 

0.4351 

Other 

730 

0.8 

0.9 

- 0.1 

0.9275 

Chose not to register for college hill time 
because could not afford to (%) 

718 

5.7 

20.8 

-15 2 *** 

0.0000 

Often or very often didn't buy textbooks 
because of the cost (%) 

727 

11.1 

23.1 

- 12.0 *** 

0.0000 

Missed class often or very often because 
needed to work (%) 

727 

5.8 

5.4 

0.4 

0.8152 

Missed class often or very often because 
couldn't pay for transportation (%) 

729 

3.7 

5.7 

- 2.0 

0.2130 

Had few or no concerns about ability to pay 
for college (%) 

714 

49.4 

36.1 

13 4 *** 

0.0003 

Had a job (%) 






First semester 

716 

48.8 

53.7 

-4.9 

0.1931 

Second semester 

714 

49.9 

53.1 

-3.2 

0.3950 

Among those who worked, 
average hours worked per week 






First semester 

359 

27.0 

28.2 



Second semester 

364 

25.4 

28.5 



Among those who enrolled, average hours per 






week spent preparing for class 
First semester 

685 

13.1 

13.8 



Second semester 

595 

13.9 

13.7 



Survey sample size 

742 

384 

358 




(continued) 
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Table 3.8 (continued) 

SOURCE: MDRC calculations from the MDRC student survey. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and cohort. 

Italic type indicates nonexperimental data. Significance tests are not calculated for nonexperimental 
data. 

a This category includes federal Pell Grants, New York State Tuition Program, Discovery financial 
resources, educational grants, and private grants. 

b This category includes Perkins Loans, Federal Direct Loans, and private bank loans. 


The proportion of program group and control group students who reported working in 
the first and second semesters was similar — roughly half of each group — and, among those 
who worked, the average number of hours worked was similar for the two groups. The propor- 
tion of students in the program group and control group who said they missed class often or 
very often because they needed to work was also similar. Although it seems ASAP’s financial 
benefits would reduce students’ need to work and reduce their likelihood of missing class 
because of work, the survey responses suggest otherwise for the evaluation sample. It may be 
that because the students in the evaluation sample had relatively low incomes, some students in 
both research groups needed to work to support themselves and their families. 

Overall, the MDRC student survey revealed some notable differences between the two 
research groups on financial issues. Important, ASAP altered how students financed their 
education and reduced the likelihood students chose not to register full time or buy textbooks 
because of the cost. The results suggest that ASAP’s financial resources made a big difference 
for students. In CUNY’s periodic student surveys and focus groups, as well in the MDRC 
student survey, students said that the program’s financial resources were one of the most helpful 
program components. 12 


Experiences in College 

As this chapter has explained, ASAP differed from the usual college services in many ways. 
Each previous section examined a particular component of the program. This section examines 
some aspects of students’ overall experiences in college. 


12 Linderman and Kolenovic (2012). 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 3.9 

Students' First-Year Experiences: Engagement in College 


Three-Year Impacts Report 


Outcome (%) 

Sample 

Size 

Program 

Group 

Control 

Group 

Difference 

P-Value 

Rated educational experience good or 

723 

86.1 

72.7 

13 4 *** 

0.0000 

excellent 

Integration and sense of belonging at school 11 

Low 

723 

9.1 

16.4 

j~! 3 *** 

0.0033 

High 

723 

18.7 

17.2 

1.5 

0.6052 

Had a college employee to turn to for advice 

722 

86.9 

57.7 

29 2 *** 

0.0000 

College faculty or staff often or very often 

made student feel important/valued 

721 

74.3 

43.8 

30.5 *** 

0.0000 

Had all or most services and supports needed 

to succeed 

724 

89.7 

57.0 

32 7 *** 

0.0000 

Had a good friend at school 

722 

80.0 

76.1 

4.0 

0.1990 

Highest degree student planned to earn 

No degree 

731 

3.7 

3.1 

0.6 

0.6565 

Associate's 

731 

8.5 

9.3 

-0.7 

0.7270 

Bachelor's 

731 

31.9 

32.4 

-0.5 

0.8909 

Master's 

731 

36.5 

41.9 

-5.4 

0.1331 

Professional or doctorate 

731 

19.4 

13.3 

6.1 ** 

0.0261 

Survey sample size 

742 

384 

358 




SOURCE: MDRC calculations from the MDRC student survey. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and cohort. 

a The scale is based on responses to nine questions about the sense of integration with and belonging 
to the school community. "Low" is the percentage of sample members scoring one standard deviation 
below the mean, indicating less integration and sense of belonging; "high" is the percentage of sample 
members scoring one standard deviation above the mean, indicating greater integration and sense of 
belonging. The questions are listed in Appendix B. 
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As the first row of Table 3.9 shows, 86 percent of program group students rated their 
educational experience as good or excellent after a year, compared with 73 percent of control 
group students. The difference of 13 percentage points is notable, especially given the relatively 
high level of satisfaction among control group students. 

The second outcome in Table 3.9 is a scale based on nine survey questions intended to 
measure a student’s feeling of integration and sense of belonging at school. Questions in the 
scale included “I feel that I matter to the college instructors, staff, and other students” and 
“College has the feeling of a community, where many people share the same goals and inter- 
ests.” (See Appendix B for a full list of the questions and a description of how the scale was 
created.) ASAP decreased the proportion of students who reported having a low sense of 
integration and belonging, but it did not increase the proportion at the high end of the scale. 

Table 3.9 presents two outcomes that are focused on students’ perceptions of college 
staff. Many more program group members than control group members said that they had 
someone at the college they could turn to for advice (87 percent, compared with 58 percent), 
and many more program group members said that they often or very often felt valued by college 
faculty or staff (74 percent, compared with 44 percent). These large differences likely reflect the 
more frequent and closer contact that program group students had with advisers and possibly 
career staff and tutors, compared with control group students. Almost all program group 
students — 90 percent — reported that they had most or all of the services and supports they 
needed to succeed. Only 57 percent of control group members reported the same. This finding 
suggests that ASAP, with its array of services, largely succeeded in meeting students’ perceived 
need for support. 

The survey also asked a question about sample members’ friendships. A total of 76 
percent of control group students said they had a good friend at school; ASAP did not signifi- 
cantly alter that proportion. Given that most control group students said they had a good friend, 
it may not be surprising that the program did not have a measurable impact on student friend- 
ships. This finding, however, considered with the findings discussed earlier, suggests that 
ASAP altered students’ connections with college staff more than it altered their connections 
with other students. 

The final outcome in the table shows sample members’ responses to a question on the 
survey about the highest degree they intended to earn. At the start of the study, the program 
group and control group responded similarly to this question. After one year, more program 
group members than control group members said they intended to earn a professional or 
doctoral degree. In other words, ASAP had an effect on students’ goals. 
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Looking to the Next Chapter 

Overall, ASAP provided a markedly different experience for students from that which the usual 
college services provided. The next chapter shifts the discussion from the implementation of 
ASAP and the service contrast to the effects of the service contrast on students’ academic 
outcomes. By comparing outcomes for the program group and control group students, the 
chapter tells the story of how the large differences in services and supports translated into large 
differences in persistence in college and graduation. 
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Chapter 4 

Effects on Educational Outcomes 


This chapter presents findings on ASAP’s estimated effects on students’ academic progress and 
completion during the three years after they entered the evaluation — the three years that ASAP 
services were offered to program group members. 1 

The chapter begins by first briefly summarizing ASAP’s three-year effects. Next, de- 
tails of these effects are provided with respect to enrollment, credit accumulation, and gradua- 
tion, followed by an assessment of ASAP’s effects for select student subgroups. Finally, 
considerations of what may occur in the future are proposed. 


Summary of Findings 

Students offered ASAP achieved much greater academic success compared with students 
offered the usual college services. Table 4.1 summarizes ASAP’s effects on three key academic 
outcomes after three years. Recall that ASAP’s goal is to graduate students within three years. 
Graduating in three years is a substantial hurdle for students who begin college with develop- 
mental education needs, an eligibility requirement for this study. Nationally, only about 15 
percent of students with developmental needs attending two-year colleges earn a certificate or 
degree in this time frame. 2 

• ASAP substantially increased students’ likelihood of persisting in school. 


'Unless specifically noted, analyses presented in this chapter include the full sample of study participants. 
Analyses reflect the estimated effect of being randomly assigned to participate in ASAP. This estimated effect 
is not the same as the estimated effect of participating in ASAP. The analyses are “intent-to-treat” as described, 
in principle, by Bloom (1984). Some individuals assigned to the program group did not participate in any part 
of ASAP (a very small number) or only participated in some parts or in some semesters. They are all included 
in the program group for the analyses. Thus, the estimand is the effect of being assigned to ASAP reflects the 
real world, where adherence to program requirements and receipt of services are imperfect. Conducting 
analyses on the full sample ensures a comparison of two groups that were similar at the outset of the study. For 
ease of exposition, 'ASAP’s effect” or “the effect of ASAP” are used in this chapter to refer to the estimated 
effect of being assigned to ASAP. 

2 These data are based on a computation of beginning postsecondary students data from the U.S. Depart- 
ment of Education’s National Center for Education Statistics (NCES) using the NCES QuickStats website 
(http://nces.ed.gov/datalab/quickstats). This statistic refers to the percentage of students who earned a certifi- 
cate or degree anywhere through 2006, among students whose first institution level in 2003-2004 was a two- 
year college and who took any remedial courses in 2004. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 


Table 4.1 

Three- Year Summary Table 
Three-Year Impacts Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

P- Value 

Sessions enrolled (out of 12) 

6.6 

5.4 

1.2 

*** 0.0000 

Total credits earned 

47.7 

39.0 

8.7 

*** 0.0000 

Earned a degree from any college (%) 

40.1 

21.8 

18.3 

*** 0.0000 

Sample size (total = 896) 

451 

445 




SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) and 
National Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Estimates are adjusted by site and cohort. 

Enrollment is based on courses in which students are still enrolled as of the end of the 
add/drop period. 


During the three-year follow-up period composed of 12 sessions of school, 3 program 
group members enrolled in 1.2 more sessions than did their control group counterparts. This 
difference represents a 22 percent increase over the control group base of 5.4 sessions enrolled. 

• ASAP had a large positive effect on total credit accumulation. 

After three years, control group students earned an average of 39.0 total credits. Pro- 
gram group students earned an average of 47.7 total credits. The difference of 8.7 total credits 
represents a 22 percent increase in credit accumulation over three years. 

• Enrollment and credit accumulation during intersessions played a key 
role in ASAP’s effectiveness. 

Compared with their control group counterparts, students in the program group were 
much more likely to enroll during intersessions, especially during the first two years of the 


’Sessions are defined in Box 4.1. 
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program. Moreover, 28 percent of the overall effect on credit accumulation occurred during 
these often overlooked sessions. 

• ASAP dramatically increased students’ likelihood of graduating within 
three years of entering the program. 

Over three years, 40 percent of program group members graduated compared with 22 
percent of control group members for an effect on graduation rates of 18 percentage points. 
Effects of this magnitude on graduation rates are unprecedented to date in MDRC’s randomized 
trials at community colleges. 

• ASAP increased students’ likelihood of enrolling at a four-year college 
within three years of entering the program. 

Not only do more ASAP students earn an associate’s degree, but more of them continue 
on to four-year institutions. 


Overall Program Effects on Academic Outcomes 

Enrollment 

Figure 4.1 (and Appendix Table C.l) displays enrollment rates at any CUNY college 
during the first three years after students were randomly assigned. Each year is split into two 
semesters, and each semester is divided into a main session and a shorter intersession. (See Box 
4.1 for a description of main sessions and intersessions.) The main session of the second 
semester serves as an example of how to understand the information presented in Figure 4.1. 
During this session, 90.3 percent of program group members (the black bar) and 80.7 percent of 
control group members (the white bar) enrolled at a CUNY college. The difference of 9.6 
percentage points (displayed above the two bars) represents ASAP’s estimated effect on 
enrollment rates during that session. 4 

The overall pattern of enrollment (for the full sample) is that students enroll at much 
higher rates during main sessions compared with intersessions, and enrollment rates decrease 
over time as students drop out or in later semesters graduate. 


4 A generalized linear model is used to estimate all effects presented in this report. The estimation model 
includes college-by-cohort covariates. The probability of being assigned to the program group was the same 
across all college-by-cohort combinations. Consequently, the estimator of the pooled average effect weighs 
each college -by-cohort-specific effect estimate by its sample size. 
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Figure 4.1 

Three- Year Enrollment at CUNY Colleges 
Three-Year Impacts Report 



SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and research cohort. 

Enrollment is based on courses in which students are still enrolled as of the end of the add/drop period. 

"Main" represents the main session for the semester. "Inter" represents the intersession for the semester. 



Box 4.1 

Timing of Academic Semesters 

Most of the analyses in this report combine outcomes for the study’s two cohorts of students, 
relative to when students enrolled in the study. For the first cohort, the first semester of the 
follow-up period refers to spring 2010, the second semester refers to fall 2010, and so on. For 
the second cohort, the first semester refers to fall 2010, the second semester refers to spring 
2011, and so on. 

At two of the three colleges in the evaluation, the academic calendar includes a fall semester 
and a spring semester. Both fall and spring are composed of two sessions: a 12-week session 
followed by a 6-week session. In this report, the longer first part of the semester is referred to 
as the “main session,” and the shorter second part is referred to as the “intersession.” The 12- 
week sessions tend to have higher enrollment rates than the 6-week sessions. Flowever, the 6- 
week sessions at these colleges typically have higher enrollment rates than winter and summer 
sessions at other colleges. 

The third college follows a more traditional academic calendar, with a 16-week fall semester, a 
3-week winter intersession, a 16-week spring semester, and two 6-week summer sessions. The 
fall and spring semesters have much higher enrollment rates than winter and summer. In order 
to discuss comparable time periods across colleges, the analyses in this report combine the fall 
and winter semesters into “fall” and combine the spring and both summer sessions into 
“spring.” (If a student attended any classes at another CUNY college, their semesters are 
handled the same way.) In the report, this college’s fall and spring semesters are referred to as 
main sessions, and the winter and summer as intersessions. 

The table below shows how the semesters are defined in the report for each cohort of sample 
members. 




2010 

2011 


Spring Semester 

Spring Summer 

Session Session 

Fall Semester 
Fall Winter 

Session Session 

Spring Semester 

Spring Summer 

Session Session 

Spring 2010 

First semester 

Second semester 

Third semester 

Cohort 

Main session Intersession 

Main session Intersession 

Main session Intersession 

Fall 2010 
Cohort 


First semester 

Main session Intersession 

Second semester 
Main session Intersession 
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ASAP consistently increased students’ likelihood of enrolling at CUNY colleges, espe- 
cially during the first two years of the program. During the main sessions of the second through 
sixth semesters, ASAP’s effects on enrollment rates were 9.6, 7.7, 9.5, 6.6, and 4.6 percentage 
points, respectively. 5 

ASAP’s effects on intersession enrollment are even more dramatic, peaking at 25.2 per- 
centage points during the second semester. 6 The large effects on intersession enrollment during 
the first two years may reflect several factors: (1) control group students enrolled at low rates, 
leaving a large margin for improvement; (2) ASAP requires full-time enrollment (attempting 12 
credits or more) and the intersession credits count toward full-time enrollment; and (3) ASAP 
advisers guided students to enroll during intersessions. The magnitude of the effects in the third 
year (during both main sessions and intersessions) is lower than in previous years. This decrease 
partially reflects the fact that by that time, many more ASAP students had earned an associate’s 
degree compared with their control group counterparts (described later in detail). 

Compared with the usual college services, ASAP clearly improves students’ likelihood 
of continuing to enroll in college. The program group’s enrollment rates during the first few 
main sessions reflect a high rate of retention. Between the first and second semesters, the 
program group’s enrollment rate dropped only 6.2 percentage points. Between the second and 
third semesters, the drop in the enrollment rate was 14.5 percentage points, a much more typical 
decrease. While ASAP had significant positive effects on persistence, developing additional 
strategies to retain students at the end of their first year could be one place to focus efforts to 
improve enrollment rates. 

Recall that to enter the MDRC evaluation, students had to be willing to enroll full time. 7 
Part-time attendance is often described as a risk factor for community college students and, as 
noted in Chapter 1, is negatively associated with academic success. Appendix Table C.l shows 
full-time enrollment at any CUNY college during the six semesters of follow-up. From the very 
first study semester, students offered the opportunity to participate in ASAP were more likely to 
enroll full time, compared with their control group counterparts (on average 12 percentage 
points more likely across the six semesters of follow-up). 


5 For the final estimate of 4.6 percentage points, p > 0. 10. 

6 For the final two intersessions, the effect estimates p > 0.10. 

throughout this chapter, full-time enrollment is defined as attempting 12 credits or more in a semester. 
As explained in Box 4.1, a semester includes both the main session and intersession. Full-time enrollment 
status is based on total credits attempted in both the main session and intersession. For the purposes of ASAP, 
Kingsborough and LaGuardia Community Colleges define full time in the way it is presented in this chapter; 
Borough of Manhattan Community College, however, does not include the summer or winter intersessions 
when determining full-time status. 
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This study found that ASAP’s combination of requirements and supports successfully 
increased full-time enrollment rates. The study, however, was not designed to disentangle which 
component(s) of ASAP caused this effect. The effect on full-time enrollment is likely the 
collective result of requiring students to enroll full time to remain in a desirable program, 
providing multiple sets of supports (financial, advising, and so forth) to enable students to meet 
this requirement, and encouraging students to enroll in intersessions. This result demonstrates 
that some students who currently enroll part time, if given the right set of requirements, incen- 
tives, and supports, would enroll full time. 

Overall, during the first three years after students entered the study, which also repre- 
sents the duration of ASAP services, the program had substantial effects on persistence in 
college. 8 


Credits Earned 

Table 4.2 depicts average credits earned at CUNY colleges during the first three years 
after students were randomly assigned. The first panel focuses on total credits earned, which 
includes both developmental (or remedial) and college-level credits. 9 The second panel includes 
developmental credits only. The third panel highlights college-level credits only. Credits are 
shown marginally by session and semester, as well as cumulatively at the end of three full years. 
Similar results are provided with additional details in Appendix Tables C.2 (credits attempted) 
and C.3 (credits earned). 10 Figure 4.2 displays total cumulative credits earned by session, and 
Appendix Tables C.4 (credits attempted) and C.5 (credits earned) provide semester-by-semester 
results for cumulative credits in detail. 


s Appendix Table C.l displays enrollment rates at any CUNY college, as presented in Figure 4.1, with 
greater detail. Appendix Table C.l also presents enrollment rates at any college covered by the National 
Student Clearinghouse (NSC), which includes data on a comprehensive set of colleges throughout the United 
States. The NSC data are important to consider since some students transfer to non-CUNY colleges. For their 
part, the CUNY data allow for a useful breakdown of enrollment by intersessions and for calculation of full- 
time attendance, unlike the NSC data. Examining both data sources provides a comprehensive picture of 
enrollment patterns for the study sample. As expected, where comparable, enrollment at any NSC college is 
higher than enrollment at CUNY colleges alone — differences are quite small during the first four semesters. 
Where comparable, ASAP’s estimated effects on enrollment at any college are very similar to those at CUNY 
colleges alone. (The greatest difference is 1.1 percentage points.) These data provide useful context when 
interpreting the credit accumulation results presented in the following section, which only includes credits 
earned at CUNY colleges. 

developmental credits are credits associated with developmental reading, writing, and math courses; 
English as a Second Language classes; and a small number of other non-college-level courses. CUNY refers to 
these credits as “equated credits.” These credits do not count towards a degree, but they do count towards 
financial aid. 

10 These tables provide additional information recommended by The What Works Clearinghouse Reporting 
Guide for Study Authors. 
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Developmental Education Students 

Table 4.2 

Three-Year Credits Earned 


Three-Year Impacts Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 


P-Value 

Total credits earned 

Semester 1 

11.4 

9.3 

2.1 

*** 

0.0000 

Main session 

9.4 

7.9 

1.5 

*** 

0.0000 

Intersession 

2.0 

1.4 

0.6 

*** 

0.0001 

Semester 2 

10.1 

7.9 

2.2 

*** 

0.0000 

Main session 

8.1 

6.7 

1.4 

*** 

0.0001 

Intersession 

2.0 

1.1 

0.8 

*** 

0.0000 

Semester 3 

8.9 

7.2 

1.7 

*** 

0.0002 

Main session 

7.3 

6.1 

1.3 

*** 

0.0011 

Intersession 

1.6 

1.1 

0.5 

*** 

0.0009 

Semester 4 

7.6 

6.1 

1.6 

*** 

0.0007 

Main session 

6.4 

5.2 

1.2 

*** 

0.0026 

Intersession 

1.2 

0.8 

0.4 

*** 

0.0034 

Semester 5 

5.5 

4.9 

0.6 


0.1766 

Main session 

4.9 

4.3 

0.5 


0.1367 

Intersession 

0.6 

0.6 

0.0 


0.9121 

Semester 6 

4.3 

3.8 

0.5 


0.2027 

Main session 

3.8 

3.4 

0.4 


0.2345 

Intersession 

0.5 

0.5 

0.1 


0.3871 

Cumulative total credits earned" 1 

47.7 

39.0 

8.7 

*** 

0.0000 

Main session 

40.0 

33.7 

6.3 

*** 

0.0002 

Intersession 

7.9 

5.5 

2.4 

*** 

0.0000 

Developmental credits earned 

Semester 1 

2.9 

1.7 

1.1 

*** 

0.0000 

Main session 

2.2 

1.5 

0.7 

*** 

0.0001 

Intersession 

0.7 

0.2 

0.5 

*** 

0.0000 

Semester 2 

1.1 

0.8 

0.3 

** 

0.0226 

Main session 

0.7 

0.7 

0.0 


0.8361 

Intersession 

0.4 

0.1 

0.3 

*** 

0.0000 

Semester 3 

0.5 

0.4 

0.1 


0.4247 

Main session 

0.4 

0.3 

0.0 


0.7803 

Intersession 

0.1 

0.1 

0.1 


0.2447 
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Table 4.2 (continued) 


Outcome 

Program 

Group 

Control 

Group 

Difference 

P-Value 

Semester 4 

0.2 

0.5 

-0 3 *** 

0.0024 

Main session 

0.2 

0.4 

-0 2 *** 

0.0089 

Intersession 

0.0 

0.1 

- 0.1 * 

0.0773 

Semester 5 

0.1 

0.2 

- 0.1 ** 

0.0184 

Main session 

0.1 

0.2 

- 0.1 ** 

0.0280 

Intersession 

0.0 

0.0 

0.0 

0.3850 

Semester 6 

0.0 

0.2 

-0 2 *** 

0.0003 

Main session 

0.0 

0.1 

_q y *** 

0.0007 

Intersession 

0.0 

0.0 

0.0 * 

0.0869 

Cumulative developmental credits earned d 

4.8 

3.8 

Y o *** 

0.0004 

Main session 

3.6 

3.3 

0.3 

0.2204 

Intersession 

1.3 

0.6 

0 7 *** 

0.0000 

College-level credits earned 





Semester 1 

8.5 

7.6 

0 g *** 

0.0065 

Main session 

7.2 

6.4 

Q g *** 

0.0055 

Intersession 

1.3 

1.1 

0.1 

0.2723 

Semester 2 

9.0 

7.1 

Y 9 *** 

0.0000 

Main session 

7.4 

6.0 

Y 4 *** 

0.0000 

Intersession 

1.6 

1.0 

0.5 *** 

0.0001 

Semester 3 

8.4 

6.8 

Y 5 *** 

0.0002 

Main session 

7.0 

5.7 

Y 2 *** 

0.0009 

Intersession 

1.4 

1.0 

q 4 *** 

0.0019 

Semester 4 

7.4 

5.6 

Y g *** 

0.0000 

Main session 

6.2 

4.9 

Y 4 *** 

0.0003 

Intersession 

1.2 

0.7 

q 4 *** 

0.0004 

Semester 5 

5.4 

4.7 

0.7 * 

0.0837 

Main session 

4.8 

4.1 

0.7 * 

0.0615 

Intersession 

0.6 

0.6 

0.0 

0.8027 

Semester 6 

4.3 

3.7 

0.6 * 

0.0919 

Main session 

3.8 

3.3 

0.5 

0.1221 

Intersession 

0.5 

0.4 

0.1 

0.2239 

Cumulative college-level credits earned a 

42.9 

35.2 

7 7 *** 

0.0000 

Main session 

36.4 

30.4 

6.0 *** 

0.0002 

Intersession 

6.6 

4.9 

Y 7 *** 

0.0001 

Sample size (total = 896) 

451 

445 
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Table 4.2 (continued) 


SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Estimates are adjusted by site and cohort. 

a Measures of cumulative credits earned exclude courses that are passed more than once. 
However, measures of cumulative credits earned in main sessions or intersessions do not 
exclude courses passed more than once. 


In terms of total (developmental and college-level) credits earned, ASAP had a positive 
effect on marginal credits earned during each of the first eight sessions (two years) of follow-up. 
In other words, during each session, program group students earned on average more credits 
than control group students. Cumulatively after three years, students offered ASAP earned 8.7 
more credits than their control group counterparts. 11 Program group members earned on average 
47.7 credits; control group members earned on average 39.0 credits. ASAP students earned on 
average 22 percent more credits than their control group counterparts. Most college courses are 
worth three or four credits. Thus students in ASAP completed on average around two to three 
more courses than their control group counterparts. 

Linking these findings back to the enrollment results, around 28 percent of the estimat- 
ed impact on cumulative total credits earned occurred during intersessions, highlighting the 
importance of these often overlooked sessions. 

During the first two semesters of study, ASAP had large effects on progress in devel- 
opmental coursework. (See developmental credit measures in Table 4.2). This finding may 
reflect ASAP’s emphasis on taking developmental courses early, and ASAP’s requirement for 
students to see tutors for their developmental courses. Starting in the fourth semester, control 
group students began to catch up with respect to developmental credit accumulation. This 
finding is largely due to the fact that many ASAP students had completed their developmental 
coursework. Nonetheless, after three years, ASAP students earned 1.0 developmental credit 
more than their control group counterparts. 

Moreover, after three years, 74 percent of program group members had completed their 
developmental education requirements, compared with 55 percent of control group members. 


U A 95 percent confidence interval on this estimated effect spans the range of 4.8 to 12.5 total cumulative 
credits earned. In terms of an effect size, the impact on cumulative credits earned is 0.29 in effect size units. 
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Total credits earned 


Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Figure 4.2 

Three- Year Total Credits Earned at CUNY Colleges 


Three-Year Impacts Report 


50.0 

45.0 
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Year 1 

Year 2 

Year 3 


SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; 
** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and research cohort. 

"Main" represents the main session for the semester. "Inter" represents the intersession for the semester. 



(See Table 4.3 for more details.) ASAP’s large effect on completing developmental education 
requirements began in the first semester of study (17 percentage point effect), and was main- 
tained over three years. 

With respect to college-level credits, the program’s effects are relatively small (although 
still positive and significant) during the first semester when ASAP had its largest effects on 
progress in developmental education. In the second, third, and fourth semesters, ASAP’s effects 
on college-level credits earned were sizable, between 1.6 and 1.9 credits. During the fifth and 
sixth semesters, program group students continued to outpace their control group counterparts 
with respect to college-level credits earned, although more modestly. Cumulatively after three 
years, ASAP students earned on average 7.7 more college-level credits than their control group 
counterparts, representing 14 percent of the 60 college-level credits required to earn a degree. 


Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 4.3 

Three Year Completion of Developmental Requirements 
Three-Year Impacts Report 


Outcome (%) 

Program 

Group 

Control 

Group 

Difference 

P- Value 

Completed developmental requirements 2 





Semester 1 

46.4 

29.9 

16.5 *** 

0.0000 

Semester 2 

63.9 

41.5 

22.4 *** 

0.0000 

Semester 3 

71.0 

48.7 

22.2 *** 

0.0000 

Semester 4 

73.8 

53.5 

20.4 *** 

0.0000 

Semester 5 

74.5 

55.3 

jq 2 *** 

0.0000 

Semester 6 

75.2 

57.1 

Jg J H=H=H= 

0.0000 

Sample size (total = 896) 

451 

445 




SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and cohort. 

“Completion of developmental requirements is contingent upon passing CUNY Assessment 
Tests, passing the highest level of developmental education, passing a college-level class in each 
subject, or both. This measure includes students who passed CUNY Assessment Tests prior to 
the first semester. 
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Degree Receipt and Transfer to Four-Year Colleges 

Table 4.4 provides summary information on degree receipt and enrollment in four-year 
colleges at any college covered by the National Student Clearinghouse (NSC). Two years after 
the study began, 8.7 percent of control group members had earned a degree and 14.7 percent of 
program group members had earned a degree. The difference of 6.0 percentage points repre- 
sents the effect of ASAP on two-year degree completion. 

In the three-year follow-up period, ASAP nearly doubled students’ likelihood of earn- 
ing a degree, increasing graduation rates by 18.3 percentage points (an 84 percent increase). 12 
By the three-year mark, program group members graduated at a rate of 40. 1 percent and control 
group members graduated at a rate of 21.8 percent. As shown in Table 4.4, nearly all of these 
degrees are associate’s degrees. 

In terms of numbers rather than percentages, among the 45 1 study participants who 
were randomly assigned to the program group, it is estimated that around 98 students (0.218 
multiplied by 451) would have earned a degree within three years had they been offered the 
usual college services. As a result of ASAP, an estimated 84 more students graduated (0.183 
multiplied by 451), for a total of 1 8 1 graduates. 13 

Although ASAP’s effect on three-year graduation rates is 18.3 percentage points, 
ASAP’s effect on earning 60 college-level credits or more — the number of credits typically 
required to earn a degree — is only 1 1 .4 percentage points (not shown in tables). 14 There are 
several possible explanations why ASAP’s effect on degree completion is 7.0 percentage points 
greater than ASAP’s effect on earning 60 college-level credits or more. 15 


12 The academic outcomes of program group members presented in this report and other MDRC reports 
from the ASAP random assignment evaluation will not align exactly with the academic outcomes for ASAP 
students presented in reports from CUNY’s internal evaluation of ASAP because the groups of students 
examined differ somewhat. First, MDRC’s outcomes reflect the academic progress of all students who are part 
of the random assignment study. During the first study semester, 16 program group students did not enroll in 
college and 7 enrolled in college but not in ASAP. These students are included in MDRC’s analyses because 
they are critical for maintaining the integrity of the experiment. CUNY’s evaluation, in contrast, only includes 
students who were enrolled in ASAP as of CUNY’s official enrollment census date. Second, ASAP serves 
some undocumented immigrants. Because those students are not eligible for federal financial aid, they are not 
eligible for the ASAP tuition waiver. Inasmuch as MDRC’s evaluation was designed to study the effects of the 
full package of ASAP services, undocumented immigrants were not included in the MDRC research sample. 
Those students are included in CUNY’s analysis. 

l3 This estimate assumes that ASAP did not cause any students who otherwise would have graduated not to 
graduate. 

14 This number includes college-level credits earned before random assignment. The estimated effects on 
degree completion include degrees earned at any college, whereas credit accumulation occurred at CUNY 
colleges only. Only one student earned a degree outside of CUNY. 

15 A very small part of this misalignment may have to do with data limitations. 
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Developmental Education Students 

Table 4.4 

Three-Year Degree and Transfer to Any Four-Year College 
Three-Year Impacts Report 


Outcome (%) 

Program 

Group 

Control 

Group 

Difference 

P-Value 

Earned a degree from any college 3 

Semester 1 

0.0 

0.0 

0.0 

— 

Semester 2 

0.2 

0.0 

0.2 

0.3175 

Semester 3 

2.9 

1.1 

1.7 * 

0.0640 

Semester 4 

14.7 

8.7 

6.0 *** 

0.0053 

Semester 5 

29.5 

15.3 

14 2 *** 

0.0000 

Semester 6 

40.1 

21.8 

18.3 *** 

0.0000 

Highest degree earned 

Certificate 

0.0 

0.0 

0.0 

— 

Associate's degree 

39.5 

21.8 

17 7 *** 

0.0000 

Bachelor's degree or higher 

0.7 

0.0 

0.7 * 

0.0832 

Enrolled in a four-year college 

Semester 1 

0.2 

0.7 

-0.4 

0.3174 

Semester 2 

0.7 

0.9 

-0.2 

0.7072 

Semester 3 

1.6 

1.3 

0.2 

0.7675 

Semester 4 

5.1 

5.4 

-0.3 

0.8448 

Semester 5 

17.1 

12.5 

4.6 * 

0.0519 

Semester 6 

25.1 

17.3 

7 g *** 

0.0040 

Sample size (total = 896) 

451 

445 




SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) and 
National Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical 
significance levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and cohort. 

a Degree receipt is cumulative. Those who earned a degree in an earlier semester are counted 
as having a degree in subsequent semesters. 


First, ASAP may successfully get students to take the right combinations of credits in 
order to earn a degree — not all combinations of credits meet degree requirements. ASAP’s 
significant additional academic advising may have informed students’ decisions. ASAP’s early 
registration policies may also have contributed by enabling students to get into the courses they 
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needed to graduate. Moreover, ASAP may have affected students’ choice of major or the timing 
in which their major was determined. 

Second, ASAP may help students who have completed the requirements to earn an as- 
sociate’s degree to take the final steps necessary to make the degree official (for example, 
completing fonns and paperwork). Data show that 9.2 percent of control group members had 
earned more than 60 college-level credits and yet did not earn a degree. In contrast, only 5.1 
percent of program group students had earned more than 60 college-level credits and yet did not 
earn a degree. The 4.1 percentage point difference could reflect ASAP helping students take the 
right combination of courses to earn a degree, assisting them with the final steps required to 
make a degree official, or both. 

The last panel of Table 4.4 presents enrollment in four-year colleges and universities 
covered by the NSC. During the sixth semester of study, one in four program group students 
was enrolled at a four-year college compared with one in six control group students. ASAP is 
not only helping more students earn associate’s degrees; many of these students are continuing 
on to four-year institutions. 

When program developers initially conceived ASAP, they had in mind the ambitious 
goal to graduate 50 percent of participants within three years of starting the program — a goal 
that CUNY’s internal analyses confirm was met for many ASAP cohorts. Important, ASAP’s 
target was set at a time when ASAP served only students without developmental course 
requirements. In contrast, MDRC’s evaluation of ASAP included a sample of 896 students, the 
vast majority of whom required one or two developmental courses. The ambitious goal of 
achieving a 50 percent graduation rate should be considered in this light. 

While ASAP may not have achieved the goal of a 50 percent graduation rate at these 
three colleges for these cohorts, the program’s effects are dramatic and unparalleled when 
compared with other programs in higher education that MDRC has evaluated using experi- 
mental methodology on a large scale. 


Subgroup Analyses 

In addition to examining the overall average effect of ASAP, the study also measured whether 
the program was effective for various types of students. ASAP’s effects were estimated sepa- 
rately by gender, whether students had earned a high school diploma before enrolling in the 
study, the number of developmental courses students needed at baseline, and students’ entry 


63 



cohort (spring or fall). 16 A fonnal test was also conducted to assess the statistical significance of 
differences in ASAP’s effects between groups (for example, whether ASAP was more effective 
for women than for men). 

Table 4.5 presents cumulative credits earned over three years for different subgroups. 
For example, after three years, women in the program group earned 48.9 credits, whereas 
women in the control group earned 40.4 credits. Thus, ASAP is estimated to have increased 
three-year credit accumulation for women by 8.5 total credits on average. Men in the program 
group earned on average 45.6 credits after three years, whereas men in the control group earned 
36.8 credits after three years. Thus, ASAP’s estimated effect on three-year credit accumulation 
for men is on average 8.8 credits. The asterisks (denoting statistical significance) and small p- 
value for estimated effect indicate that ASAP was beneficial to both women and men. 

In terms of credit accumulation, ASAP was effective for all of the study’s subgroups. 
The last column of the table provides the p-value for a test of differential effects: for example, it 
tests for evidence that the estimated effect for women (8.5 total credits) was different from the 
estimated effect for men (8.8 total credits). Even though ASAP was effective for both women 
and men, it is theoretically possible it could have been more effective for one gender than the 
other. No significant evidence indicates that ASAP was more effective for any of the study’s 
subgroups compared with another. 

Table 4.6 presents similar subgroup infomiation with respect to degree completion after 
three years. In temis of degree completion, ASAP’s effects are positive and statistically signifi- 
cant for all subgroups examined — men, women, those with and without a high school diploma 
at baseline, those with zero or one developmental education need at baseline, those with two or 
more developmental education needs at baseline, and fall and spring cohorts. 17 


16 There was no strong a priori hypothesis why ASAP would be more effective for any particular subgroup. 
ASAP’s comprehensive services would seem potentially beneficial to most types of students. These groups 
were explored (with the exception of the entry cohort) because the average outcomes of the groups tend to vary 
in community colleges. For example, students with fewer developmental requirements tend to outperform 
students with more developmental requirements. Additional subgroup analyses of interest were not conducted 
owing to data restrictions. 

17 With respect to the subgroups investigated, no clear evidence indicates that ASAP was more effective 
for any one of these groups. It is worth noting that this evaluation’s sample size is not large enough to detect 
differential effects among subgroups that many people would consider practically significant. For example, 
there is a nearly 9 percentage point difference in ASAP’s estimated three-year graduation effects for students 
without a high school diploma (11.6 percentage points) and those with a high school diploma (20.5 percentage 
points). If this difference were real, it would be noteworthy. However, since the estimated effects for each 
group are not very precise (there were only 227 sample members without a high school diploma at baseline), 
an estimated difference in effects of 9 percentage points could easily occur by chance, even if the program was 
exactly equally effective for both groups. Thus, what is noteworthy with regards to the subgroup findings is the 
evidence that ASAP had large positive effects for every specified subgroup (gender, high school diploma status 
at baseline, and developmental courses needed at baseline). 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 4.5 


Total Credits Earned After Three Years: 
Variation in Effects by Student Characteristics 


Three- Year Impacts Report 


Student Characteristic 

Sample 

Size 

Average Credits Earned 
Program Control 
Group Group Difference 

P- Value 
for 

Difference 

P-Value for 
Differential 
Estimated 
Effects 

Gender 






0.9519 

Female 

556 

48.9 

40.4 

8.5 *** 

0.0005 


Male 

340 

45.6 

36.8 

g g *** 

0.0075 


Sample size 

896 






Earned high school 







diploma at baseline 3 






0.8003 

No 

227 

44.0 

36.2 

7.8 * 

0.0504 


Yes 

669 

48.9 

40.0 

g g *** 

0.0001 


Sample size 

896 






Number of developmental 







courses needed at baseline 






0.4353 

l b 

331 

52.4 

43.9 

8.6 *** 

0.0049 


2 C 

466 

44.3 

39.0 

5.4 * 

0.0535 


Sample size 

797 






Cohort 






0.5543 

Spring 2010 cohort 

327 

48.9 

38.7 

10.2 *** 

0.0016 


Fall 2010 cohort 

569 

47.0 

39.2 

7 g *** 

0.0016 


Sample size 

896 







(continued) 


Acceleration and Control Group Catch-Up 

The preceding sections describe ASAP’s positive effects on important academic outcomes for a 
variety of student types over three years. ASAP’s effect on earning a degree within three years is 
large, at over 18 percentage points. That said, when considering the potential benefits of ASAP, a 
perspective beyond three years is important. For example, consider the two following hypothet- 
ical scenarios. In one scenario, the control group manages to catch up to the program group in the 
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Table 4.5 (continued) 


SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences of impacts between subgroups. Statistical significance 
levels are indicated as: fff = 1 percent; f f = 5 percent; f = 10 percent. For the measures presented in 
this table, no statistically significant differences between subgroups were observed. 

Estimates are adjusted by site and cohort. 

a Students shown as not having a high school diploma include those who earned no degrees, those 
who earned a General Educational Development (GED) certificate, and those who are missing degree 
information. Students shown as having a high school diploma are those who earned a high school 
diploma, an occupational or technical certificate, or another, unspecified higher degree. 

b This group includes a small number of students who had no developmental need at baseline (about 5 
percent of the 33 1 students). 

c This group includes a small number of students who needed three or more developmental courses at 
baseline (about 14 percent of the 466 students). 


fourth year after random assignment with respect to graduation rates. Such a finding would imply 
that ASAP enabled students who would have graduated in four years to graduate more quickly. 
(That is, it accelerated graduation.) In another scenario, the graduation effect remains the same or 
grows in the fourth year after random assignment and beyond. Such a result would mean that 
ASAP enabled students who otherwise would never have graduated to earn a degree. Accelerat- 
ing the time it takes students to earn a degree (first scenario) is an important, noteworthy accom- 
plishment that could result in significant benefits (financial and otherwise). However, the 
potential benefits of enabling students who otherwise would never have graduated to earn a 
degree are even larger. Long-term follow-up on the study sample will provide the best evidence 
on this matter. In the meantime, it may be useful to examine students’ final academic status at the 
three-year mark in more detail in order to speculate what may come in the near future. The 
current evidence suggests that ASAP enabled many students to graduate who otherwise would 
never have graduated. Moreover, although the control group may begin to catch up over time, 
such catching up is unlikely to be substantial within the next year of follow-up. 

Table 4.7 presents students’ final status at the end of the current follow-up period. The 
second panel divides holders of an associate’s degree into three categories based on their 
enrollment status during the last semester of the three-year follow-up period. Notably, 20.6 
percent of the program group earned an associate’s degree and were enrolled in a four-year 
college during the final semester of study, compared with 1 1 .2 percent of control group mem- 
bers. Time will tell whether this difference will grow or shrink, and whether it might lead to 
significant effects on earning a bachelor’s degree. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 4.6 

Degrees Earned at Any College After Three Years: 
Variation in Effects by Student Characteristics 


Three- Year Impacts Report 








P- Value for 



Earned a degree (%) 


P- Value 

Differential 


Sample 

Program 

Control 


for 

Estimated 

Student Characteristic 

Size 

Group 

Group Difference 

Difference 

Effects 

Gender 






0.7752 

Female 

556 

41.6 

24.3 

Y! 3 *** 

0.0000 


Male 

340 

37.3 

18.2 

IQ j *** 

0.0001 


Sample size 

896 






Earned high school diploma 
at baseline 3 






0.1870 

No 

227 

31.3 

19.7 

11.6 ** 

0.0467 


Yes 

669 

43.1 

22.5 

20.5 *** 

0.0000 


Sample size 

896 






Number of developmental 
courses needed at baseline 






0.3902 

l b 

331 

48.7 

27.0 

2\ 7 *** 

0.0001 


2 ° 

466 

35.5 

19.5 

16.0 *** 

0.0001 


Sample size 

797 






Cohort 






0.2154 

Spring 2010 cohort 

327 

44.0 

20.8 

23 3 *** 

0.0000 


Fall 2010 cohort 

569 

37.8 

22.4 

15.5 *** 

0.0001 


Sample size 

896 







(continued) 


Some students in both the program and control groups earned an associate’s degree and 
did not enroll during the last semester of study, 10.0 percent of the program group compared 
with 4.7 percent of the control group. For these students, the associate’s degree may have been 
terminal. Recall that ASAP’s effect on enrollment rates, while still positive, was at its lowest 
during the final semester. (In fact, the effect was not statistically significant.) Part of the expla- 
nation may be that many more program group members had earned an associate’s degree by the 
three-year mark and subsequently ended their studies. 
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Table 4.6 (continued) 


SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) and National 
Student Clearinghouse data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences of impacts between subgroups. Statistical significance 
levels are indicated as: fff = 1 percent; f f = 5 percent; f = 10 percent. For the measures presented in 
this table, no statistically significant differences between subgroups were observed. 

Estimates are adjusted by site and cohort. 

a Students shown as not having a high school diploma include those who earned no degrees, those 
who earned a General Educational Development (GED) certificate, and those who are missing degree 
information. Students shown as having a high school diploma are those who earned a high school 
diploma, an occupational or technical certificate, or another, unspecified higher degree. 

b This group includes a small number of students who had no developmental need at baseline (about 5 
percent of the 331 students). 

c This group includes a small number of students who needed three or more developmental courses at 
baseline (about 14 percent of the 446 students). 


Might the Control Group Catch Up? 

The third panel of Table 4.7 is most relevant to the possibility that the control group 
might catch up to the program group with respect to degree attainment. The first row shows that 
at the three-year mark 59.9 percent of program group members had not earned a degree, com- 
pared with 78.2 percent of control group members (a difference of 18.3 percentage points). Just 
over half of this effect (-9.3/-1 8.3) came from students who had not earned a degree and were not 
enrolled during the final semester of study. While it is theoretically possible that students who 
had not earned a degree and did not enroll in the final semester could return to school and earn a 
degree, most likely they will not. Thus, it is probable that ASAP’s effect on earning an associ- 
ate’s degree will remain above 9 percentage points, at a minimum, in the long term. 

Important to the possibility of the control group catching up, 7.0 percentage points more 
control group students without a degree were enrolled in a two-year college during the final 
semester of study. This gap may represent some potential for the control group to catch up since 
these students were continuing to make progress toward an associate’s degree. Of course, not all 
students who enrolled during the final study semester are equally likely to graduate in the short 
term. For example, it is reasonable to expect that only students with 48 college-level credits or 
more are within reach of completing a degree in the next semester. In total, 12.6 percent of the 
control group appeared to be within reach of earning an associate’s degree within one semester. 
This group included those students who had not yet earned a degree, were enrolled at a two-year 
college during the final study semester, and had earned more than 48 college-level credits by the 
end of three years. In the program group, 8.7 percent of students met those same criteria. The 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 4.7 

Status at Three Years 


Three- Year Impacts Report 


Outcome (%) 

Program 

Group 

Control 

Group 

Difference 

P-Value 

Bachelor's degree earned 

0.7 

0.0 

0.7 * 

0.0832 

Associate's degree earned 

39.5 

21.8 

17 7 *** 

0.0000 

Enrolled in four-year (during final semester) 

20.6 

11.2 

q 4 *** 

0.0001 

Enrolled in two-year (during final semester) 

8.9 

5.8 

3.0 * 

0.0803 

Not enrolled (during final semester) 

10.0 

4.7 

5 2 *** 

0.0027 

Has not earned a degree 

59.9 

78.2 

-18.3 *** 

0.0000 

Enrolled in four-year (during final semester) 

4.0 

6.1 

-2.1 

0.1599 

Enrolled in two-year (during final semester) 

17.3 

24.3 

.7 o ** 

0.0100 

Earned 48 college-level credits or more 

8.7 

12.6 

-3.9 * 

0.0583 

Earned 36-47 college-level credits 

3.3 

5.4 

-2.1 

0.1259 

Not enrolled (during final semester) 

38.6 

47.9 

_g ^ *** 

0.0049 

Sample size (total = 896) 

451 

445 




SOURCE: MDRC calculations from CUNY Institutional Research Database (1RDB) data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by site and cohort. 

Enrollment is based on courses in which students are still enrolled as of the end of the add/drop 
period. 

Number of college-level credits includes credits earned before random assignment. 


3.9 percentage point difference suggests that the control group might have the potential to catch 
up slightly in the next semester. 18 Although it is possible that over time the control group will 
significantly catch up to the program group’s rate of degree attainment, it does not appear likely 
that this scenario will occur in the next one or two semesters. 


18 Notably, using data from the first through fifth semesters, it is possible to apply a similar approach to 
project graduation impact estimates in the second through sixth semesters, when the impact estimates are 
known. Results from such projections are only moderately successful, and the projections are the least accurate 
when projecting from the fifth to sixth semester. 
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ASAP has had a dramatic effect on the academic achievement of a variety of different 
low-income student populations. Compared with the usual college services at the three-year 
mark, students offered ASAP enrolled at higher rates, accumulated credits at higher rates, 
graduated at higher rates, and enrolled in four-year colleges at higher rates. This comprehensive 
program is clearly improving students’ academic success. A reasonable next question is “at 
what cost?” The next chapter addresses that question. 
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Chapter 5 

Cost-Effectiveness of ASAP 


This chapter analyzes the cost and resources that the City University of New York (CUNY) and 
its partners invested in the ASAP program in relation to the estimated effect of the program. 
First, it identifies the cost of ASAP, including the costs of the components associated with its 
comprehensive services. Next, it compares the total cost of college for ASAP students with the 
total cost for students receiving the usual college services. Last, it compares the total cost of 
college and student academic outcomes for both groups in order to explore how the ASAP 
program changed the cost per outcome achieved — specifically, whether the investment in the 
program produced more graduates within three years per dollar than the usual college services. 1 


Summary of Findings 

The key findings are the following: 

• The direct cost of the ASAP program is $14,029 per program group 
member over three years. This estimate includes $6,238 on administration 
and staffing, $2,927 on student services, $1,558 on course enrollment, and 
$3,305 on financial supports. 

• When additional costs of educating students are considered, over the 
three years of follow-up, the college invested $16,284 more per ASAP 
group member than it did per control group member. This estimate in- 
cludes the direct cost to operate ASAP ($14,029) plus an estimate of the cost 
associated with ASAP students attempting more college courses during that 
time ($2,256). The program increased the investment in ASAP students by 
63.2 percent above the $25,781 spent on the typical student receiving the 
usual college services. On an annual basis, the program increased the invest- 
ment in the average program group member by roughly $5,428. 

• The cost-effectiveness analysis of ASAP highlights its ability to lower the 
cost per degree earned within three years. The analysis shows that the 


'it is possible to lower the cost per outcome achieved while increasing total costs. The analysis compares 
alternative strategies to achieve a designated goal and the result is always comparative, that is, one program 
lowers the cost per outcome achieved when compared with another. In this case, the program is compared with 
the usual college services. Each alternative requires resources to produce its respective result. The comparison 
aims to highlight which strategy or alternative produces the most positive outcomes per dollar of investment. 
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$16,284 of additional investment in each ASAP program group student re- 
sulted in an estimated 83 .9 percent increase in the likelihood of earning a de- 
gree. Even though ASAP spent more money overall, this estimated effect ac- 
tually lowered the cost per degree earned for ASAP students by 1 1 .4 percent 
compared with students who receive the usual college services. 


Methodology 

This chapter describes the cost of the ASAP program and estimates the cost-elfectiveness of 
two cohorts of ASAP students at three CUNY colleges (Borough of Manhattan, Kingsborough, 
and LaGuardia Community Colleges) from spring 2010 through summer 20 13. 2 All dollar 
values in this analysis have been adjusted to 2013 dollars using the Higher Education Price 
Index for public two-year colleges. Costs are associated with the steady state operation of 
ASAP. All costs reflect the costs and resources of operating ASAP for the two cohorts at the 
three colleges in the study for the three-year evaluation period. All costs associated with 
MDRC’s evaluation of ASAP have been excluded. Costs are considered from the perspective of 
the college. College-level spending includes resources from a spectrum of stakeholders, 
including students, private donors/foundations, local taxpayers, and state and federal govern- 
ments. These costs are estimated using college financial infomiation. Since all funds (such as 
tuition paid by students, subsidies from various governments, and private donations) are 
funneled through the college, this approach provides a good estimate of the total investment that 
was made in these community college students. For presentation purposes, all costs have been 
classified as direct costs, base costs, or indirect costs. See Box 5.1 for definitions of terms used 
in this chapter. 


Direct Cost 

The direct cost of the program accounts for the resources that are required to operate ASAP 
for the 451 program group students during the period of program evaluation at 
the three participating colleges. Table 5.1 illustrates the direct cost of ASAP, which was 
an estimated $14,029 per program group member over the three-year period. Hiese values are 
based on CUNY’s expenditure data on ASAP and adjusted using the Higher Education Price 


‘At two of the schools, two cohorts were randomly assigned, one in spring 2010 and one in fall 2010; at 
the third school, only one cohort was randomly assigned, in fall 2010. 
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Box 5.1 


Key Cost Terms 

The following ter ms are used in the cost analysis. 

Direct cost: The cost directly associated with providing ASAP, including components such 
as administration and staffing, student services, course enrollment supports, and financial 
supports. 

Base cost: The cost of the usual college services in the absence of the program. Base cost = 
cost per credit x number of credits attempted by the control group. The cost per credit is an 
estimate of the average amount of resources expended by the college to provide one credit 
of study; it is calculated by dividing the college’s annual operating budget by the number of 
credits attempted at the college during the year of interest. 

Indirect cost: The cost resulting from a behavioral change brought about by the program, 
such as additional credits attempted by program group members. This cost can extend 
beyond the period of program operation. The cost per credit for indirect cost is based on the 
college’s annual operating budget excluding academic support and student services. Indi- 
rect cost of the program = cost per credit (excluding academic support and student ser- 
vices) x additional credits attempted by program group members. 

Program group cost: The total cost of educating program group members over three years 
of follow-up. Program group cost = direct cost + base cost + indirect cost. Program group 
cost can be divided by the number of program group members to get the cost per program 
group member. 

Control group cost: The total cost of educating control group members over three years of 
follow-up. Control group cost = base cost. Control group cost can be divided by the 
number of control group members to get the cost per control group member. 

Net cost: The cost difference between program group members and control group mem- 
bers. Net cost = program group cost - control group cost. 

Cost-effectiveness analysis: An evaluation in which the net costs of alternative interven- 
tions are expressed as the cost per unit of a desired outcome. In this analysis, cost- 
effectiveness is presented for cost per degree earned and cost per total credit earned. 

Cost per degree earned: The amount invested in the research group of interest per degree 
earned by that research group. For the program group, cost per degree earned = program 
group cost -r the percent of program group members who earned a degree. 

Cost per credit earned: The amount invested in the research group of interest per credit 
earned by that research group. For the program group, cost per credit earned = program 
group cost -r number of total credits earned by the program group members. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) 
for Developmental Education Students 

Table 5.1 

Direct Cost of ASAP per Sample Member 


Three- Year Impacts Report 


Program Component 

Cost ($) 

Percentage of Total (%) 

Administration and staffing 

Administration 

3,247 

23.1 

Research and evaluation 

1,152 

8.2 

Other 

1,840 

13.1 

Subtotal 

6,238 

44.5 

Student services 

Advising 

1,754 

12.5 

Career and employment services 

569 

4.1 

Tutoring 

604 

4.3 

Subtotal 

2,927 

20.9 

Course enrollment 

Blocked or linked courses 

1,363 

9.7 

ASAP seminar 

195 

1.4 

Subtotal 

1,558 

11.1 

Financial supports 

MetroCards 

1,749 

12.5 

Textbooks 

1,106 

7.9 

Tuition waiver 

451 

3.2 

Subtotal 

3,305 

23.6 

Total Direct Cost 

14,029 

100.0 


SOURCE: MDRC calculations based on CUNY's ASAP expenditure data and the Higher Education 
Price Index. 

NOTES: Tests of statistical significance were not performed. 

Rounding may cause slight discrepancies in sums and differences. 

Program costs are based on a steady state of operation that excludes external research and start- 
up costs. 

All costs are shown in constant 2013 dollars. 


Index. 3 Administration and staffing is the largest spending category; it includes $3,247 on 
program administration, $1,152 on research and evaluation, and $1,840 on other expenses. 


3 The direct costs by year are as follows: first year ($6,599), second year ($4,536), and third year ($2,895). 
The cost per program group member is greatest during the first year and decreases with time primarily because 

(continued) 
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Student services accounted for $2,927 (20.9 percent); it includes $1,754 on advising, $569 on 
career and employment services, and $604 on tutoring. Course enrollment accounted for $1,558 
(11.1 percent); it includes $1,363 for course blocks and $195 (1.4 percent) for the ASAP 
seminar. Financial supports accounted for $3,305 (23.6 percent); it includes $1,749 for Metro- 
Cards, $1,106 for textbooks, and $451 for tuition waivers. A detailed description of each 
spending category is listed below. 

Administration and Staffing 

• Administration covers the cost of administrative staff at the individual col- 
leges and ASAP’s central office who run and manage ASAP throughout the 
CUNY system. It includes the director of the program, program coordinator, 
clerical staff, and a small percentage of costs associated with fiscal services. 

• Research and evaluation covers the cost of CUNY’s internal and ongoing 
evaluation of the ASAP program. It pays for two full-time employees and in- 
formation management systems associated with institutional capacity build- 
ing, internal program evaluation, and program improvement. It does not in- 
clude costs associated with the MDRC evaluation of ASAP. 

• Other covers the cost of the student leadership program, office supplies, pro- 
fessional development, consultants, travel costs, marketing materials, com- 
puters, and an indirect institutional charge specific to the CUNY Research 
Foundation and Office of Academic Affairs. 

Student Services 

• Advising covers the cost of ASAP advisers. ASAP advisers had smaller 
caseloads per adviser (60 to 80 students compared with the norm of 600 to 
1,500 students per adviser). These small caseloads allowed advisers to meet 
with ASAP students more frequently and for longer periods of time. 

• Tutoring covers the cost of ASAP tutoring services. ASAP students in de- 
velopmental courses or on academic probation are required to attend academ- 
ic tutoring lessons separate from the normal college services. 


enrollments decline with time, so the first year covers the cost of most students. Even if the cost per full-time 
student remained constant over time, the cost per sample member would still decline because some sample 
members drop out over time. Additionally, the ASAP seminar only occurs within the first few semesters, after 
which ASAP students are not offered the service. 
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Career and employment services covers the cost of ASAP-specific career 
and employment specialists who advise students on jobs, career planning, 
scholarships, and networking, and who host career fairs. 


Course Enrollment 

• Blocked or linked courses covers the cost of reserving seats for ASAP stu- 
dents in courses that are scheduled back to back or as a set in their first year. 4 

• ASAP seminar covers the cost of the required seminar course that ASAP 
students took within their first few semesters. 

Financial Supports 

• Tuition waiver covers the cost of waiving the difference between an ASAP 
student’s tuition and fees and financial aid (the sum of the federal Pell Grant 
and New York State Tuition Assistance Program grant). 

• MetroCards covers the cost of providing free unlimited monthly Metro- 
Cards to ASAP students who meet program requirements. 

• Textbooks covers the cost of ASAP’s buyback/rental policy with the college 
bookstore, which provides free use of textbooks to ASAP students. 

Controlling Costs 

ASAP has expanded since its inception in the CUNY system. As ASAP has grown, it 
has been able to control per-student costs by achieving economies of scale, leveraging relation- 
ships, and targeting ASAP services to high-need students. The program has achieved economies 
of scale by spreading fixed costs across a greater number of ASAP students. For example, the 
cost of administration per student generally decreases as programs expand. The program has 
leveraged relationships by negotiating discounted prices with the Metropolitan Transit Authori- 
ty and Bames and Noble bookstores to lower the cost of MetroCards and textbooks, respective- 
ly, for ASAP students. Specifically, ASAP paid on average $73 per monthly MetroCard over 


4 Blocked or linked courses, along with the costs associated with the tuition waiver, make up what is 
known as the “faculty replacement” fee within the CUNY system. This fee was designed as a lump sum paid 
out to the colleges by CUNY Central in exchange for the waiving of ASAP students’ tuition (the portion that 
was not covered by financial aid), reserving spots or blocking entire courses for ASAP students, and occasion- 
ally opening extra summer or winter session courses for ASAP students. For the purposes of this analysis, the 
faculty replacement fee was split to show the amount associated with the tuition waiver and costs that went to 
the colleges to reserve and block courses for ASAP students. 
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the three-year follow-up period, while the cost to an individual student was $112 at the end of 
the follow-up period. ASAP also developed a textbook buyback policy, which provides students 
with the free use of textbooks at a much lower cost than if ASAP were to buy textbooks for 
each student. Finally, program participants are low-income students who generally receive 
financial aid that largely covers the cost of tuition. As a result, the cost of the tuition waiver for 
ASAP students has been one of the smallest program components (roughly 3 percent of 
ASAP’s direct costs). If ASAP were expanded to include students who do not receive as much 
financial aid, then it is possible that the costs associated with the tuition waiver could increase, 
potentially increasing the cost of the program dramatically. 


Base Cost 

In order to understand the context of ASAP’s direct cost, this analysis estimates the cost of the 
usual college services provided to non-ASAP students. This cost is referred to as the base cost. 
This analysis uses the estimated cost of credits attempted as a proxy for base costs. This 
approach assumes that the use of resources corresponds to the number of credits attempted. In 
other words, a student who attempts more credits is generally associated with greater expendi- 
tures than a student who attempts fewer credits. 5 Credits attempted serves as an appropriate 
driver of base costs because it provides a simple measure of the level of a student’s engagement 
with the college. To estimate the dollar value of credits attempted, the number of credits 
attempted is multiplied by an estimated cost per credit. This cost per credit is estimated by 
dividing the college’s annual total expenses and deductions by total instructional activity (credit 
and contact hours attempted) at the college during the year of interest as reported in the financial 
and enrollment data on the National Center for Education Statistics’ Integrated Postsecondary 
Education Database System (IPEDS). (These values include the cost of depreciation and the 
cost of scholarships.) 6 The average cost per credit, in constant 2013 dollars, yielded by this 
calculation is then used to estimate the cost of the usual college experience. 7 This approach is 
not perfect. One limitation is the assumption that all credits attempted have the same cost to the 


5 It is possible, however, that students use services such as advising and counseling independent of the 
number of courses in which they enroll. 

depreciation accounts for the cost of long-lived assets such as buildings, equipment, and vehicles. Alt- 
hough such assets are not consumed in one year, they are consumed with time. Including depreciation ensures 
that the cost of this consumption is accounted for. For more information on accounting standards in IPEDS, 
both for depreciation and scholarships, see the IPEDS factsheet, “IPEDS Finance Data FASB and GASB — 
What’s the Difference?” (http://nces.ed.gov). 

7 Total expenses and deductions, as reported on IPEDS for public institutions in the new Governmental 
Accounting Standards Board (GASB 34/35) form, are divided by IPEDS’s enrollment measure of total 12- 
month instructional activity (reported in credit or contact hours) by year to calculate the cost per credit 
attempted at a college. This cost per credit at the three CUNY schools in the study was calculated to be a 
weighted average of $470.53 in constant 2013 dollars. 
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college, which is not the case. 8 For example, science lab courses may be more expensive than 
English courses. In order to use this approach, the analysis assumes that the average cost of a 
CUNY student is representative of the average cost of a student in the sample. 9 These assump- 
tions seem reasonable for this analysis because the process of random assignment helps ensure 
that any differences in the cost of credits attempted compared with the average cost likely 
occurs similarly in both the program and control groups. Estimating the base cost helps show 
how much money is spent to educate the typical student at CUNY in the absence of ASAP. 

Table 5.2 presents all of the costs used in this cost analysis, and calculates a net cost per 
group member. The first row reiterates the direct cost that was discussed earlier. The second 
row of Table 5.2 presents the base cost of credits attempted at the colleges in the study. Control 
group members attempted an average of 54.8 credits in the three years of follow-up, for a total 
cost of credits attempted (credits attempted multiplied by cost per credit) of $25,781 per group 
member. This represents the cost of the typical college experience at the CUNY study schools 
in the absence of ASAP. 


Indirect Cost 

Indirect cost is estimated based on the number of additional credits attempted by ASAP students 
compared with control group students. This analysis uses two approaches: an upper bound 
based on average costs excluding the cost of academic and student services and a lower bound 
in which indirect costs equal zero. An average of these two approaches will be carried forward 
as the primary estimate of indirect costs. 

Indirect cost equal to the average cost (excluding the cost of academic and student ser- 
vices) represents the case in which the college is unable to absorb the cost of additional credits 
attempted because existing resources are already fully leveraged and new resources are re- 
quired. For example, if students are enrolling in additional courses that are filled to capacity, 
then the college may have to open new course sections, which would require an increase in 
resources. This cost per credit is different from the cost per credit used in the base cost estimate. 
The cost per credit for the base cost is estimated using the college’s total expenditures and total 
instructional activity in credit hours. The cost per credit for indirect cost is estimated similarly, 
except that the cost of academic and student services is excluded from the colleges’ expenses. 10 


8 “Cost” in this case refers to the amount of resources dedicated to the course by the college; it is not neces- 
sarily connected to the price that students may be paying for that course. 

9 As discussed in Chapter 2 and shown in Table 2.1, the evaluation sample is relatively similar to the target 
population. 

10 The colleges’ total instructional activity in credit hours is unchanged. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 5.2 

Net Cost of Education per Sample Member 


Three- Year Impacts Report 


Feature ($) 

Program 

Group 

Control 

Group 

Difference 

(Net) 

Direct cost: cost of primary program components 

14,029 

0 

14,029 

Base cost: cost of credits attempted in the absence of the program 

25,781 

25,781 

0 

Indirect cost: cost of additional credits attempted due to the program 

2,256 

0 

2,256 

Upper bound: marginal cost equal to average cost a 

4,511 

0 

4,511 

Lower bound: marginal cost equal to zero’ 3 

0 

0 

0 

Total cost 

42,065 

25,781 

16,284 

Upper bound: marginal cost equal to average cost a 

44,320 

25,781 

18,540 

Lower bound: marginal cost equal to zero’ 3 

39,809 

25,781 

14,029 


SOURCE: MDRC calculations based on student level participation data, program-specific budget 
data, and financial and enrollment data from the Integrated Postsecondary Education Data System. 


NOTES: Tests of statistical significance were not performed. 

Rounding may cause slight discrepancies in sums and differences. 

Program costs are based on a steady state of operation that excludes external research and start- 
up costs. 

Credits attempted include all college-level and developmental credits attempted. 

“"Marginal cost equal to average cost" represents the case in which existing college resources 
cannot be leveraged to accommodate changes in credits attempted, therefore incurring additional 
costs to the college. The additional cost to the college, or the marginal cost of the additional credits 
attempted, is approximated as the average cost per credit attempted at the institution, excluding the 
cost of academic support and student services which ASAP is already providing. 

b "Marginal cost equal to zero" refers to the ability of existing college resources to absorb the 
cost of additional credits attempted by the program group without incurring new costs to the 
college. 


The reason for this difference is to avoid double counting the costs associated with the academic 
and student services since the ASAP program already pays for additional student services. 11 The 
indirect cost estimate sets costs equal to the midpoint between the upper bound (when indirect 


"The calculation of base cost for ASAP students did not make a similar exception since all students have 
access to the college’s usual services and those in the program could have used these services in addition to the 
services provided by ASAP. 
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cost is equal to the average cost per credit excluding academic and student services) and the 
lower bound (when indirect cost is equal to zero). 

The third section of Table 5.2 shows the indirect cost of the program associated with 
program group members attempting more credits than control group members. Program group 
members attempted more credits than their control group counterparts throughout the three 
years. On average, each program group member attempted 66.3 credits by the end of the follow- 
up period, approximately 11.5 credits more than the average control group member. Multiply- 
ing the additional 1 1.5 credits attempted by the corresponding average cost per credit (excluding 
academic and student services) gives an indirect cost of additional credits attempted by the 
program students of $4,5 1 1 per program group member. 

The lower-bound estimate sets the indirect cost per credit equal to zero in order to cap- 
ture a condition in which colleges are fully able to absorb the cost of additional credits attempt- 
ed without additional cost. This estimate would be reasonable if most of the additional credits 
students attempted were large lecture classes in which some seats were open. In a university 
setting, it is possible that the marginal cost of additional attempted credits equals the average 
cost per credit or the marginal cost of additional attempted credits equals zero. Moreover, a 
university generally does not open new course sections until a certain threshold of interest is 
reached (otherwise students wait until the next semester to take a closed course). Therefore, the 
analyses use the estimate that sets the indirect cost equal to the midpoint (or average) between 
the conditions when the marginal cost of additional attempts is equal to average cost (excluding 
the cost of academic and student services) and when marginal cost is equal to zero. According- 
ly, the indirect cost of additional credit attempts by program group members is $2,256. The 
conclusions of the cost-effectiveness analysis remain the same regardless of which indirect cost 
value is used. 


Net Cost 

Net cost is defined as the difference between the program group cost and the control group cost. 
The costs of each group are presented in the total line of Table 5.2. The total cost is calculated 
by adding the direct cost, base cost, and indirect cost. The total cost of ASAP per program 
group member over the course of the program was $42,065, compared with the $25,781 cost to 
educate the average control group member. Over three years, the net cost is $16,284 per 
program group member. 12 


12 This cost analysis presents the average cost of ASAP per program group member. This estimate spreads 
costs across all students who started in the program group, including those who enrolled less than full time, 
dropped out, or graduated. Cost results are described using this approach (rather than using a cost per full-time 

(continued) 
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Cost-Effectiveness Analysis 

A cost-effectiveness analysis expresses the cost of alternative interventions as the cost per unit 
of a desired outcome. 13 This cost-effectiveness analysis considers the cost per degree earned and 
the cost per credit earned. These estimates spread costs across all students who started in the 
program group, including those who enrolled less than full time, dropped out, or graduated. 
Table 5.3 summarizes the results. The top row presents the cost per group member. (These 
values were described in the preceding “Net Cost” section.) The total cost per program group 
member ($42,065) is $16,284 more than the total cost per control group member ($25,781). The 
second row shows the percentage of program and control group members who earned a degree 
within three years. Specifically, it shows that 40.1 percent of program group members and 21.8 
percent of control group members earned a degree after three years. Therefore, under normal 
conditions at the three CUNY community colleges, the cost per degree earned for the evaluation 
sample was $1 18,248. ASAP lowered the cost per degree earned to $104,825. Although ASAP 
costs an additional $16,284 per program group member, the program was cost-effective in 
terms of degrees produced. Specifically, the 18.3 percentage point increase in earning a degree 
was large enough to lower the cost per degree earned by $13,423 (1 1.4 percent). The findings of 
this analysis corroborate the findings of a 2012 study on ASAP’s cost-effectiveness conducted 
by Levin and Garcia, which estimates that ASAP lowered the cost per degree by 14.7 percent. 14 

A cost-effective intervention reduces the cost per outcome compared with the status quo 
in such a way that its proportional effect on the desired outcome is greater than or equal to the 
proportional change in cost. In this case, ASAP increased the likelihood of earning a degree by 
18.3 percentage points, from 21.8 to 40.1 percent. This change represents an 83.9 percent 
increase in the likelihood of earning a degree. This increase is proportionately more than the 
63.2 percent increase in the cost associated with ASAP. 15 

A similar cost-effectiveness analysis was perfonned in relation to total credits earned. It 
showed that the cost per credit earned for the program group is higher than the cost per credit 


equivalent approach) in order to best align cost estimates with the outcomes and effects that are described in 
Chapter 4. 

13 For additional explanation of this approach, see Sommo, Mayer, Rudd, and Cullinan (2012). A similar 
approach is also used in Levin and Garcia (2012). 

14 Levin and Garcia (2012). The differences in the cost per graduate values in the Levin and Garcia analy- 
sis and this analysis are caused by multiple factors including: different graduation rates, different levels of 
enrollment, and different sources for full-time equivalent (FTE) cost estimates. That is, the MDRC estimate is 
based on data available in IPEDS (which includes the cost of depreciation and some types of scholarships), 
whereas the Levin and Garcia analysis uses a cost per FTE used by the CUNY budget office (which excludes 
the cost of depreciation and some types of scholarships). 

l5 This finding means ASAP could spend an additional $5,387 per student and remain cost-effective in 
terms of lowering the cost per degree earned. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Table 5.3 

Cost-Effectiveness Values 
Three-Year Impacts Report 


Outcome 

Program 

Group 

Control 

Group 

Difference 

(Impact) 

Cost per group member ($) 

42,065 

25,781 

16,284 

Earned a degree (%) 

40.1 

21.8 

18.3 *** 

Cost per degree earned ($) 

104,825 

118,248 

-13,423 

Total credits earned 

47.7 

39.0 

g 7 *** 

Cost per credit earned ($) 

882 

661 

221 

Sample size (total = 896) 

451 

445 



SOURCE: MDRC calculations from program-specific participation and budget data, transcript data, and 
financial and enrollment data from Integrated Postsecondary Education Data System (IPEDS). 

NOTES: Rounding may cause slight discrepancies in sums and differences. All dollar values have been 
rounded to the nearest whole dollar. 

Program costs are based on a steady state of operation that excludes external research and start-up 
costs. 

Tests of statistical significance have only been performed on outcome measures not costs. All 
outcomes are cumulative over three years. For these measures, a two-tailed t-test was applied to 
differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** 
= 5 percent; * = 10 percent. Estimates are adjusted by cohort-campus interaction, National Center for 
Education Statistics risk factors, and pre-random assignment placement test scores. 


earned for the control group. The last section of Table 5.3 shows the total credits earned per 
program and control group member: 47.7 and 39.0, respectively. However, the proportional 
change in credits earned, an increase of 22.3 percent (8.7 credits in addition to a base of 39.0), is 
less than the proportional change in cost (an increase of 63.2 percent). Specifically, as shown in 
the last rows of Table 5.3, the cost per credit earned for the program group is $882 compared 
with the control group cost per credit earned of $661, a difference of 33.4 percent. Therefore, 
ASAP was unable to lower the cost per total credits earned. 16 


16 The fact that ASAP lowered the cost per outcome in terms of completing a degree within three years but 
not in terms of total credits earned reinforces the importance of the relationship between the percentage 
increase in cost and the percentage change for an outcome. For example, the percentage increase in costs 
associated with ASAP incurred over the normal college experience is 63.2 percent. Meanwhile, the 8.7 
increase in total credits earned by program group students is a 22.2 percent change over control group students’ 

(continued) 


82 






Cost-effectiveness analyses only highlight the relative efficiency of a program com- 
pared with the status quo without the program. 17 Having a program that is cost-effective 
compared with the status quo does not necessarily mean that its economic benefits exceed its 
cost nor does failing to have a program appear cost-effective compared with the status quo 
mean that its economic benefits fail to exceed its costs. While this analysis does not estimate the 
economic value of a two-year degree and hence does not estimate the economic value of 
completion, other research has. In particular, a 2013 benefit-cost analysis of ASAP conducted 
by Levin and Garcia indicates that the value of a two-year college degree easily exceeds the cost 
of producing degrees, 18 suggesting that producing more degrees results in economic gains. 
Specifically, for each dollar of investment in ASAP by taxpayers, the return was between $3 
and $4, and around $12 for each dollar invested by the individuals, suggesting that ASAP is a 
very productive public and private investment. 


Conclusion 

This analysis shows that the program invested an additional $16,284 in the average program 
group member. This additional investment resulted in an 18.3 percentage point increase in 
degree completion within three years and an 8.7 increase in total credits earned over the control 
group. It is important to track these students for a longer time period for several reasons. First, 
the control group could catch up, as described in Chapter 4. Second, even if the program’s 
estimated effect remains the same and the levels of both groups go up, the cost-per-outcome 
analysis could still switch directions. For example, if the control group eventually graduates at 
50 percent and the program group at 68 percent, this could change the program’s relative 
efficiency in tenns of degrees earned. 19 However, even if the program impact fades over time 
because the control group catches up, the program benefits could still be greater than the costs 
since benefits accrue during the period for which an impact exists. 

After three years, ASAP’s impact on earning a degree is large enough that the cost per 
degree earned by program participants is actually lower than the cost per degree earned by 
students receiving the usual college services. Specifically, the cost per degree earned for the 


total credits earned. This 22.2 percent change is not enough to account for the 63.2 percent increase in cost, and 
therefore ASAP does not lower the cost per outcome with regard to total credits earned. 

17 Cost-effectiveness values are one way to evaluate the performance of ASAP relative to its cost. There 
are other frames that could produce different results. For example, a benefit-cost analysis of the program would 
estimate the value of increased earnings and other benefits to see if the benefits of the program exceed the cost 
of the program. 

ls Levin and Garcia (2013). 

19 Relative cost-efficiency could change because the percentage improvement on degree receipt would be 
substantially lower while the percentage increase on program cost would still be relatively large. Since this 
cost-effectiveness frame compares the proportionate changes, the results of the analysis could change. 
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program group is $13,423 (11.4 percent) less costly than the cost per degree earned by the 
control group. This difference is important because it shows that while more resources were 
invested in the program group, the impact of ASAP on earning a degree within three years was 
large enough to lower the cost per degree for ASAP students compared with students receiving 
the usual college services. 
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Chapter 6 


Implications of the Findings 
and the Future of ASAP 


This report shares some great news in a field in which good news can seem rare. MDRC’s 
evaluation found that the ASAP program was well implemented at the three participating City 
University of New York (CUNY) community colleges, and its requirements were enforced 
during the period studied. The program provided an intensive array of supports over three years, 
and the difference was substantial between ASAP and the usual college services available to the 
study’s control group. MDRC’s study, like CUNY’s internal study, found that ASAP was very 
successful in improving students’ academic outcomes. Over three years, ASAP almost doubled 
graduation rates of low-income community college students in need of some developmental 
education. The dramatically higher graduation rates caused the cost per graduate to fall, despite 
the incremental costs of operating the program. Overall, ASAP’s effects after three years are the 
most positive MDRC has found in over a decade of research in higher education. 

This chapter offers some thoughts on which program components may be most im- 
portant in driving the effects and explains some aspects of the evaluation’s findings. The chapter 
ends with a brief description of what is next for the research and for ASAP. 


Speculation on Why ASAP Worked and 
Lessons for Other Colleges 

What drove the large effects found in the study and which of ASAP’s components were most 
important in improving students’ academic outcomes? MDRC’s evaluation was not designed to 
definitively answer that question. Ultimately, each component in ASAP had the potential to 
affect students’ experiences in college, and MDRC’s evaluation estimates the effect of ASAP’s 
full package of services on students’ academic outcomes. It may be useful, however, to explore 
evidence that suggests whether any components were more or less important in changing 
students’ outcomes — both to help interpret the effects of ASAP presented in this report and to 
inform college administrators considering a program similar to ASAP. This commentary is 
much more speculative than the findings presented earlier in the report. 

It is likely that ASAP’s full-time enrollment requirement, coupled with multiple sup- 
ports to facilitate that enrollment, were central to the program’s success. As discussed in prior 
chapters, ASAP required students to enroll in school full time, and the program increased the 
proportion of students who did so. It also increased the proportion of students who enrolled in 
winter and summer intersessions. This resulted in program group students’ accumulating more 
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credits than control group students and moving through school more quickly. It is one thing to 
advise students to enroll full time and hope that they can find the resources to do so; it is an 
entirely different thing to advise them to attend full time while also covering their tuition, books, 
and transportation, and providing an array of enhanced student services to support them in 
school. Recall that 90 percent of program group students said that they had most or all of the 
services and supports that they needed to succeed in school, and 87 percent of program group 
members said that they had someone at college they could turn to for advice. It is unknown 
exactly how much support is necessary to yield substantial effects on full-time enrollment, but it 
seems unlikely that such a requirement paired with far more limited financial and student 
service supports would yield the same impact. 

In addition to requiring students to enroll full time, ASAP requires students to partici- 
pate in various program services. During the follow-up period, students were required to see 
their adviser twice a month, meet with the career and employment specialist once a semester, 
and attend tutoring frequently if they were taking a developmental course or were on academic 
probation. To help encourage students’ engagement in key program services, ASAP linked this 
participation with receipt of a monthly MetroCard for use on public transportation. As noted 
earlier, a monthly MetroCard cost $112 at the end of the follow-up period for this report — a 
substantial cost for a low-income student. In a city where millions of people travel primarily or 
exclusively on public transportation, a MetroCard is a strong incentive and very likely increased 
students’ participation in ASAP services. 

To effectively link MetroCard receipt with participation, it was important to keep track 
of students’ engagement in the program. As described in Chapter 3, colleges closely tracked 
students’ participation in key ASAP components. This tracking allowed the program staff to 
closely monitor students’ participation, adjust advisement as needed, and distribute MetroCards 
appropriately. Along with CUNY Office of Academic Affairs’ (CUNY Central’s) other 
evaluation efforts, it also permitted the college ASAP directors and the CUNY Central ASAP 
team to monitor program implementation and identify possible areas for improvement. This 
high level of monitoring and assessment, with a focus on improvement, likely contributed to 
ASAP’s thorough implementation and its significant effects for students. 

ASAP encouraged students to take their developmental courses early. Students in the 
program group moved through their developmental courses more quickly than control group 
members, and after three years many more program group members had completed their 
developmental education requirements. 

While ASAP overall provides an uncommon degree of enhanced supports for students, 
it also has more requirements compared with the usual college services. This pairing of oppor- 
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tunity and obligation appears to be an important characteristic of the program. ASAP has high 
expectations of students in exchange for a comprehensive three-year package of services. 

Listening to the voices of students who participated in ASAP may help in assessing the 
importance of the program’s specific services and benefits. As mentioned earlier, CUNY 
Central’s ASAP staff periodically surveyed participating students and convened focus groups. 
In both the surveys and focus groups, students pointed to the financial resources and advisement 
as the most helpful program components and the ones with which they were most satisfied. 1 A 
number of students supplied additional comments on the MDRC student survey, and many said 
that the program’s financial supports and advisement were the most important elements of their 
success in school. Although it may be challenging for people to accurately identify what 
motivates complex behaviors, such as staying in school despite barriers or advancing to gradua- 
tion when the majority of students do not, the students’ assessments corroborate MDRC 
researchers’ data-based observations. The students who responded to CUNY’s surveys and 
focus groups also sometimes pointed to early registration as important to them. 2 It seems likely 
that the opportunity to register early, before seats filled up and courses closed, allowed more 
students to get the courses they needed and to create convenient schedules. 

ASAP’s financial supports were substantial and, based on the MDRC student survey, 
made a difference in students’ experiences in college. The tuition waiver may have been on the 
whole less important than the free use of textbooks and free MetroCards for the MDRC evalua- 
tion sample, since only a small proportion of the program group students received a waiver. The 
waiver, however, may have been critical to many of the students who received it: It may have 
made the difference between attending school full time, part time, or not at all. Furthermore, the 
availability of a tuition waiver as part of ASAP’s package of services may have affected even 
students who did not receive it. Some students may have been more willing to enroll in college 
(and enroll full time) because they knew the waiver was available and therefore would not need 
to worry about how they might pay for their tuition and fees. 

Chapter 3 describes dramatic differences between the quantity and content of the advis- 
ing available to program and control group students. As Chapter 4 fleshes out in more detail, 
ASAP’s impact on three-year graduation rates is larger than its impact on earning 60 college- 
level credits or more, the number of credits typically needed to earn an associate’s degree. This 
finding suggests that ASAP may have led students to take the right combination of credits to 
earn a degree or identify a major sooner, or it may have prompted students to take the final steps 
necessary, such as filing paperwork, to make the degree official. In either case, the ASAP 
adviser would have played a central role. 


'Linderman and Kolenovic (2012). 
2 Linderman and Kolenovic (2012). 
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Students in CUNY’s focus groups and surveys identified the blocked or linked courses 
as the least helpful of the ASAP components. 3 This finding also corroborates MDRC research- 
ers’ observations. As discussed earlier, that component of ASAP was implemented somewhat 
differently across the colleges, and most students in the program group did not take a complete 
block of courses. Most program group students took at least one course with a concentration of 
ASAP students (at least five students), but it seems unlikely that this component was central in 
driving the program’s substantial effects. 

It seems likely that specific students were helped by particular components at particu- 
lar times. For example, a student who was struggling in his developmental math course may 
have been substantially helped by the ASAP tutor in that class. Tutoring may have made the 
difference — for that student, in that class — between failing and passing. Another student 
may have been connected to an internship by an ASAP career and employment specialist that 
helped her learn important on-the-job skills and solidify her career choice and commitment to 
getting a four-year degree. Any of the components may have made a big difference for at least 
some students. 

That said, the effects presented in this report represent the impact of ASAP’ s full pack- 
age of services. It is unknown what the effect of a modified program would be and it seems 
likely that a less intensive program would yield less substantial changes for students. 

Overall, this exploration points to the following lessons for other colleges when design- 
ing and implementing programs and reforms: 

• Requiring full-time enrollment in college while also providing an array of on- 
going supports for students, such as enhanced advisement and financial sup- 
ports, can yield substantial changes in enrollment and credit accumulation. 

• Intersessions, perhaps especially summer, provide good opportunities to in- 
crease enrollment in college and credit accumulation. 

• Requiring students to participate in key program components, monitoring 
participation, and providing a meaningful benefit to those who participate 
fully can markedly increase receipt of services. 

• Monitoring program operations, with a focus on ongoing improvement, con- 
tributes to strong implementation. 

• Encouraging or requiring students to take developmental courses early in 
their time in college can hasten and increase completion of those courses. 


3 Linderman and Kolenovic (2012). 



Although the study did not isolate the effect of each of the practices described above, it 
provides ample evidence that they have great promise. 


Implications of the Findings 

The Promise of Comprehensive Reform 

Graduation rates for community college students are very low, especially for students 
who need developmental courses to build their basic skills. Nationwide, only about 15 percent 
of students with developmental needs attending a two-year college earn a degree or certificate 
within three years. 4 College administrators and policymakers have been seeking interventions 
that can make a difference for students. Many programs have been found to help students 
academically in the short term, but few have made significant progress toward the ultimate goal 
of college completion. 

To the authors’ knowledge, ASAP’s effects are unparalleled in large-scale experimental 
evaluations of programs in higher education to date, and policymakers and college administra- 
tors should consider implementing similar programs. In MDRC’s experimental evaluations of 
community college reforms, the next largest increase in three-year graduation rates is 4 percent- 
age points (compared with ASAP’s impact of 18 percentage points). 5 The findings in this report 
show that a comprehensive, long-term intervention can substantially boost students’ success. 
Many of ASAP’s components have been tested before on their own. ASAP, however, brings 
them together in a holistic package. By providing an array of services and supports over three 
years, ASAP targets multiple potential barriers to students’ success. These services and supports 
help a variety of students with different barriers as well as students with multiple barriers or 
different barriers over time. 

ASAP may be uncommon in the breadth and intensity of its package of services. How- 
ever, many higher education experts are calling for bold, large-scale reforms, 6 and existing 
reforms have been moving in this direction. For example, the Gates Foundation’s Completion 
by Design initiative provides colleges with funding and technical assistance to reform the 
college experience from start to finish. 7 Another example is the Lumina Foundation for Educa- 

4 These data are based on a computation of beginning postsecondary students data from the U.S. Depart- 
ment of Education’s National Center for Education Statistics (NCES) using the NCES QuickStats website 
(http://nces.edu.gov/datalab/quickstats). This statistic refers to the percentage of students who earned a 
certificate or degree anywhere through 2006, among students whose first institution level in 2003-2004 was a 
two-year college and who took any remedial courses in 2004. 

5 Patel, Richburg-Hayes, de la Campa, and Rudd (2013). 

6 Bailey (2014). 

7 www.completionbydesign.org. 
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tion’s Finish Faster initiative, which aims to help higher education systems in select states 
develop and implement guided pathways to a degree with enhanced monitoring and support. 8 
Although the effects for these initiatives have on students are not yet known, they reflect a shift 
away from short-term, narrowly focused reforms to a bolder, more comprehensive approach. 

Even with ASAP’s comprehensive services and unprecedented success in improving 
students’ outcomes, a substantial proportion of program group students left college without a 
degree. By the end of the three-year follow-up period, 39 percent of the program group had not 
earned a degree and were not enrolled in college. Some of those students might return to school 
in later semesters or might have found a good job without a degree, but the finding highlights 
the ongoing challenge of fostering college completion for low-income students. 

Considerations About ASAP’s Costs 

It may not be surprising that ASAP’s effects are large, given the comprehensive nature 
of the program, that it provides services consistently for up to three years, and that it substantial- 
ly increases the level of investment in each student. MDRC’s evaluation estimated that CUNY 
spent about $16,300 more per program group member over three years than it spent per control 
group member. However, the study also found that, because ASAP boosted graduation rates so 
dramatically, the program actually cost less per graduate than did the usual colleges services — 
at least at the three-year point. 

When considering ASAP’s cost as presented in this report, it is important to bear in mind 
that many of the costs depended on the context in which the program was operated. For example, 
the cost of living in New York City is among the highest in the nation; staff salaries and fringe 
benefits in CUNY ASAP are likely higher than those at many other colleges. The costs also 
include administration and management at the colleges and several staff at CUNY Central. Most 
other colleges, however, are not overseen by a central administrative body, and operation costs 
might be lower. Most students in the program group did not need the tuition waiver. If the 
program had targeted students who were less likely to get complete coverage from their financial 
aid, the cost would have been higher. In short, an ASAP-like program in another setting is 
unlikely to be low cost, although its total cost would depend on various factors. 

ASAP’s success in increasing students’ academic outcomes makes it a model for col- 
leges in states where funding for higher education is linked to performance outcomes. Perfor- 
mance-based funding first started in Tennessee in 1979, and to date more than half of U.S. 
states have tried it. Using indicators such as student retention, attainment of certain credit levels, 


8 See project description on the Community College Research Center’s website 
(http://ccrc.tc.columbia.edu). 
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and graduation rates, perfonnance-based funding links funding for higher education institutions 
directly to students’ performance. 9 If an ASAP-like model that was operated in another college 
could generate effects similar to those found at CUNY, the program would simultaneously 
improve students’ outcomes and secure additional funds for the college. Of course, if many 
colleges in a state adopted ASAP and it was as successful as it was at CUNY, funding formulas 
might need to be revised or additional dollars might need to be secured. 

Helping an Array of Students 

MDRC’s evaluation has shown that ASAP was highly effective for students who need- 
ed one or two developmental education courses: Students in the program group moved through 
their developmental courses more quickly, and after three years many more program group 
members than control group members had completed their developmental education require- 
ments. ASAP’s substantial effects on graduation were especially noteworthy because they were 
found for a group of students who did not have all the basic skills they needed for college-level 
courses when they entered the study. (CUNY’s internal evaluation has found substantial effects 
for college-ready students as well. 10 It is unknown what ASAP’s effects would be for students 
who needed more remediation than did the students in the evaluation sample.) 

The higher education field has been struggling to develop initiatives that substantially 
help students with developmental education needs, and ASAP is a model to consider. ASAP 
provides enhanced student services, including tutoring and financial supports, but it does not 
change the curriculum or pedagogy in developmental education classrooms. There is still work 
to be done to improve what happens inside the community college classroom — and many 
refonns are being tried — but the results from this study show that outcomes for students with 
developmental education needs can be markedly improved, and students can even graduate 
relatively quickly, with the right package of supports, requirements, and messages. 

It is notable that ASAP generated positive effects for all subgroups of students exam- 
ined, including those defined by gender, high school diploma status, and number of develop- 
mental courses needed at the start of the study. As described earlier, the majority of students in 
the evaluation sample were relatively young when they entered the study, lived at home with 
their parents, were unmarried, and did not have children. At the same time, however, the sample 
included a substantial number of students considered to be nontraditional college students: 23 
percent of the evaluation sample were 23 or older when they entered the study, 26 percent did 
not live with their parents, 31 percent were employed, 15 percent had at least one child, and at 


9 Doherty and Reddy (2013). 

10 Linderman and Kolenovic (2012). 
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least 6 percent were married. 1 1 ASAP was designed to help a broad array of students, and this 
report presents evidence that it was successful. 

ASAP requires students to enroll in college full time. Some higher education experts 
argue that many community college students simply cannot attend full time because of family 
obligations, work, or other issues. Nationwide, about 40 percent of community college students 
(roughly 2.8 million students) attend school full time. 12 This study shows that ASAP did boost 
full-time enrollment, compared with regular college services. In other words, some students 
who would have attended college part time without ASAP attended full time because of the 
program. It is unclear, however, what the effects might be with a different target group, such as 
low-income parents. It is also unclear what outcomes an ASAP -type program that did not 
require full-time enrollment would yield. 


What’s Next? 

ASAP’s effects at the three-year point are unprecedented. This section briefly describes the 
expansion of ASAP within CUNY. It also presents a further possible study using the MDRC 
evaluation sample and efforts to replicate the program at other colleges. 

Expansion of ASAP Within CUNY 

Based on results from its internal study of ASAP and the positive effects from the ran- 
dom assignment evaluation, CUNY committed to expand the program substantially. The 
current goal is to serve over 13,000 students by fall 2016 across the original six community 
colleges and additional CUNY colleges. Medgar Evers College in Brooklyn, a college in the 
CUNY system offering both associate’s and bachelor’s degrees, launched ASAP in fall 2014. 

In the expansion, most ASAP features have remained the same. In order to lower the 
program’s costs, however, there are a few changes, including fewer semesters of the ASAP 
seminar and somewhat larger caseloads for advisers. Each ASAP adviser is now responsible for 
no more than 150 students — more than during the evaluation period but still far fewer students 
than with whom non-ASAP advisers work. During the first semester, all students meet with 
their adviser twice a month. Each subsequent semester, students are placed into one of three 
needs groups based on academic, personal resiliency, and program compliance criteria. The 
groups receive differentiated types of contact (individual, group, telephone, and e-mail) with a 


n Fifteen percent of the evaluation sample did not indicate their marital status on the Baseline Information 
Form. 

12 American Association of Community Colleges’ Community College Enrollment website 
(http://www.aacc.nche.edu/AboutCC/Trends/Pages/enrollment.aspx). 
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sustained focus on maintaining strong relationships with advisers. Tracking data from CUNY 
shows that the vast majority of students continued to have contact with their adviser at least 
once a month. 

Longer-Term Evaluation of CUNY ASAP 

ASAP’s effects for the sample of students in this random assignment evaluation are 
very notable at the three-year point. It is also important to know what happens to these students 
in the longer term, and MDRC hopes to raise funds to continue studying students’ outcomes. 
ASAP was designed both to increase the proportion of students who receive a degree and to 
help them graduate more quickly. At the three-year point, ASAP may have caused students to 
graduate who would not have without the program, accelerated graduation for students who 
would have done so eventually, or both. Therefore, open questions about academic outcomes 
include the following: Do program group students continue to earn associate’s degrees at high 
rates? Do control group members begin to catch up in terms of earned associate’s degrees? 
What are the longer-term effects of ASAP on bachelor’s degrees? It is also important to know 
what happens to students in the labor market. A college degree is important in part because of 
its potential to benefit its recipients in the form of better job opportunities. Finally, the program 
was found to be cost-effective in terms of producing graduates at the three-year point. When 
considering ASAP’s costs relative to its effects for students, it is important to consider a longer 
time frame. If a program’s effect on graduation rates changes, its cost-effectiveness may also 
change. 


Replicating and Evaluating ASAP Outside of CUNY 

ASAP is a highly promising program that merits testing in other settings. It is important 
to know whether the comprehensive ASAP program can be successfully implemented by other 
colleges, in different contexts with different students, and yield substantial effects. Studying 
replications of ASAP at different colleges would help answer that question and provide infor- 
mation on how the program can be adapted for settings outside of CUNY and New York City. 
It would also help show whether ASAP is a cost-effective investment for other colleges. 

ASAP, with its unprecedented effects for students, has received much attention in the 
higher education field, and many colleges have begun exploring whether they might adopt it or 
a similar model. MDRC is working with CUNY to develop evaluations of ASAP and ASAP- 
like programs at interested colleges to learn more about what might be driving the effects and 
whether the program can successfully operate at a lower cost. If the results indicate that ASAP’s 
effects are consistently positive across institutions outside of CUNY, they may offer decisive 
evidence that the model may be effective at most other colleges. If the program’s effects vary by 
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college, the results will shed light on the kinds of colleges and students that ASAP would 
benefit most. 

Overall, MDRC’s evaluation found that CUNY developed and successfully implement- 
ed a program that generated large, meaningful changes for low-income students with develop- 
mental education needs at three urban community colleges. With an investment in the right 
combination of enhanced services, requirements, and messages, community college students 
can succeed at far higher rates than usual. The ASAP model offers a promising strategy to 
markedly increase graduation rates and build human capital among disadvantaged populations. 
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Appendix A 

Additional Baseline Information 




Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Appendix Table A.l 

Selected Characteristics of Sample Members at Baseline 
Three-Year Impacts Report 

FdT 

Outcome Sample 


Marital status (%) 

Married 6. 1 

Unmarried 78.6 

Missing 15.3 

Lives with parents (%) 73.7 

Parents pay more than half of expenses (%) 41.0 

Missing 18.0 

Has one or more children (%) 15.3 

Currently employed (%) 31.3 

Among those currently employed, number of hours worked per week in current job (%) 

I- 10 hours 8.1 

II- 20 hours 34.0 

21-30 hours 31.7 

31-40 hours 24.7 

More than 40 hours 1.5 

Highest grade completed (%) 

10th grade or lower 7.3 

11th grade 7.8 

12th grade" 75.9 

Missing 9.0 

Diplomas/degrees earned b (%) 

High school diploma 73.8 

General Educational Development (GED) certificate 20.8 

Occupational/technical certificate 5.6 

Other 1.7 

None 6.0 

(continued) 
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Appendix Table A.l (continued) 


Outcome 

Full 

Sample 

Date of high school graduation/GED receipt (%) 

During the past year 

49.4 

Between one and two years ago 

13.3 

Between two and five years ago 

13.1 

More than five years ago 

13.2 

Has not earned a diploma/GED a 

6.0 

Missing 

5.0 

Highest degree student plans to attain (%) 

Associate's 

2.8 

Bachelor's 

31.4 

Master's 

41.6 

Professional or doctorate 

17.8 

Beyond an associate's, unspecified 

6.4 

First person in family to attend college (%) 

30.3 

Highest degree/diploma earned by mother (%) 

Not a high school graduate 

19.5 

High school diploma or GED 

21.8 

Some college, did not complete a degree 

16.0 

College degree 0 

18.6 

Missing 

24.1 

Language other than English spoken regularly in home (%) 

44.7 

Sample size 

896 


SOURCE: MDRC calculations using Baseline Information Form (B1F) data. 

NOTES: Estimates are adjusted by site and research cohort. 

Missing values are only included in variable distributions for characteristics with more than 5 
percent of the sample missing. 

Distributions may not add to 100 percent because of rounding. 

Characteristics shown in italic type are calculated for a proportion of the full sample. 
a This number includes students who were enrolled in high school at study intake. 
b Distributions may not add to 100 percent because categories are not mutually exclusive. 

C A college degree includes associate's, bachelor's, master's, and doctorate degrees. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Appendix Table A.2 

Developmental Courses Needed at Baseline 


Three-Year Impacts Report 


Outcome (%) 

Full 

Sample 

Number of developmental courses needed 

None 

2.1 

1 

39.4 

2 

50.6 

3 or more 

7.9 

Subject of developmental need 

English 

16.8 

Math 

53.3 

English and math 

27.7 

Sample size 

797 


SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) data. 

NOTES: Students without pre -random assignment CUNY Assessment Test data in all subject 
areas are excluded from this table. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and cohort. 
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Appendix B 

MDRC Student Survey Documentation and Analyses 




This appendix addresses two aspects of the MDRC student survey: 


1. Survey response bias analysis: An analysis of the response rate and the potential 
for bias in the survey results. 

2. Creation of survey scales: A description of two scales created from the student 
survey and reported in Chapter 3 (the Quality of Advising scale and the Integration 
and Sense of Belonging scale). 


1. Survey Response Bias Analysis 

This section of the appendix analyses the response rate for the MDRC student survey, and the 
potential for bias in the results. 


Survey Fielding and Respondent Sample 

The MDRC student survey asked study participants about a variety of topics including their 
participation in and experience with student services, educational experiences, work experience, 
and financial situation. The survey was fielded to all 896 evaluation sample members approxi- 
mately one year after random assignment. Students in the spring 2010 cohort were surveyed 
February through May 2011. Students in the fall 2010 cohort were surveyed September through 
December 2011. A total of 742 responses were collected, equivalent to an overall survey 
response rate of 83 percent. 1 Program group students responded at a rate of 85.1 percent and 
control group students at a rate of 80.8 percent. 


Characteristics of Survey Respondents 

Three analyses were conducted to test for potential biases. First, characteristics of survey 
respondents are compared with the characteristics of students who did not respond to the 
survey. These results provide an indication of how representative the survey respondents are of 
the full study sample — a form of external validity. Second, three-year academic impacts of 
survey respondents are compared with those of nonrespondents. These results also serve as an 
indication of external validity. Finally, characteristics for program group students who respond- 
ed to the survey are compared with characteristics for control group students who responded to 


'Two students out of the study sample of 896 were excluded from the calculation of response rate. At the 
time of the survey fielding, these students were either away or unavailable to respond for the duration of the 
survey (1 student) or could not be contacted due to a language barrier (1 student). Excluding these 2 students, a 
total of 894 students remained. A total of 742 responses represent 83 percent of this group. 
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the survey — these results provide an indication of whether the results are internally valid for 
survey respondents. 

Comparison of Respondent and Nonrespondent 

Baseline Characteristics 

Appendix Table B.l compares baseline characteristics for survey respondents and non- 
respondents. The table indicates that respondents and nonrespondents were similar with regard 
to all baseline characteristics measured. However, all survey bias analyses are limited to a small 
number of baseline characteristics. 2 Thus, respondents and nonrespondents may differ on 
umneasured baseline characteristics. 

An omnibus F-test was conducted to see whether students’ baseline characteristics were 
jointly predictive of responding to the survey. 3 The F-test yielded a p-value of .20, suggesting 
that respondents and nonrespondents do not differ significantly in their measured baseline 
characteristics. This finding provides evidence that the survey results may generalize to nonre- 
spondents. 

Comparison of Respondent and Nonrespondent Academic Impacts 

Appendix Table B.2 compares the estimated impact of ASAP on key academic out- 
comes, for survey respondents and nonrespondents. For all three academic outcomes examined, 
the table shows that the estimated impact of ASAP is much larger for survey respondents than 
for nonrespondents. Among survey respondents, 24.2 percent of control group members earned 
a degree within three years, compared with 44.1 percent of program group members, for an 
estimated impact of 19.8 percentage points. Among nonrespondents, ASAP’s estimated impact 
on earning a degree is only 5.4 percentage points. The final column indicates that this difference 
in estimated impacts is statistically significant. Similar patterns are observed for enrollment and 
credit accumulation measures: ASAP’s estimated impact is much larger among survey respond- 
ents than among nonrespondents. Appendix Table B.2 suggests that survey respondents and 
nonrespondents reacted differently to ASAP. Thus, the survey results among respondents may 
not generalize to those who did not respond to the survey. 4 


2 For both analyses, data confidentiality concerns limited the number of baseline characteristics available 
for these comparisons. 

3 Logistic regression was used for this analysis in cases where the outcome was whether a sample member 
responded to the survey and the predictor variables were students’ baseline characteristics. 

4 As stated in Chapter 3, the survey is the main data source used to measure treatment contrast. If the ser- 
vice contrast is suspected to be related to academic impacts, then the analysis presented in Appendix Table B.2 
probably suggests that the survey results may be slightly larger than what they would have been had all study 
sample members responded to the survey. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Appendix Table B.l 

Characteristics of ASAP Survey Respondents and Nonrespondents 


Three-Year Impacts Report 



Number of 

Full 

Survey 

Survey 


Characteristic (%) 

Observations 

Sample 

Respondents 

Nonrespondents 

P-Value 

Female 

894 

62.1 

63.2 

56.7 

0.1426 

Earned a high school diploma or 

higher 





at baseline 3 (%) 

894 

74.6 

75.5 

70.2 

0.1823 

Number of developmental courses 





needed at baseline 





0.2459 

0 

795 

2.1 

2.3 

1.5 


1 

795 

39.2 

40.6 

32.6 


2 

795 

50.7 

49.7 

55.6 


3 or more 

795 

7.9 

7.4 

10.4 


Subject of developmental need 





0.6058 

English 

778 

17.1 

17.5 

15.0 


Math 

778 

54.5 

54.7 

53.4 


English and math 

778 

28.4 

27.8 

31.6 


Sample size 


894 

742 

152 



SOURCE: MDRC calculations using Baseline Information Form (BIF) data and CUNY Institutional 
Research Database (IRDB) test data. 

NOTES: A two-tailed t-test was applied to differences between survey response groups for gender and 
dipomas/degrees earned. A chi-squared test was applied to differences between the groups for 
developmental courses and subjects needed. Levels for statistically significant differences between 
program and control groups are indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 

To analyze whether on average survey respondents and nonrespondents differed from each other, an 
omnibus F-test was performed, which yielded a p-value of 0.1971. This finding suggests that on the 
baseline charateristics shown above, survey respondents and nonrespondents do not differ from one 
another. 

Estimates are adjusted by site and cohort. 

Missing values are not included in individual variable distributions. 

Distributions may not add to 100 percent because of rounding. 

“This number includes high school diplomas, occupational and technical certificates, and unspecified 
other types of degrees. Not included are students who earned no degree, earned a GED and no other 
degrees, or who are missing degree information. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Appendix Table B.2 

Three- Year Academic Outcomes of 
ASAP Survey Respondents and Nonrespondents 


Three- Year Impacts Report 


Outcome 

Sample 

Size 

Program 

Group 

Control 

Group Difference 

P- Value 
for 

Difference 

P-Value for 
Differential 
Estimated 
Effects 


Sessions enrolled 






0.0404 

tt 

Respondents 

742 

6.9 

5.7 

Y 3 *** 

0.0000 



Nonrespondents 

152 

4.5 

4.3 

0.2 

0.7381 



Total credits earned 






0.0483 

tt 

Respondents 

742 

50.8 

41.2 

g 6 *** 

0.0000 



Nonrespondents 

152 

29.3 

29.9 

-0.5 

0.9076 



Earned a degree from 








any college 






0.0331 

tf 

Respondents 

742 

44.1 

24.2 

Yg 3 *** 

0.0000 



Nonrespondents 

152 

17.5 

12.1 

5.4 

0.3589 



Sample size 

894 








SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) and National 
Student Clearinghouse data. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

A two-tailed t-test was applied to differences of impacts between survey respondents and 
nonrespondents. Statistical significance levels are indicated as: fff = 1 percent; f f = 5 percent; f = 10 
percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and cohort. 


While ASAP appears to be more effective for students who responded to the survey, 
ASAP’s estimated effects for survey respondents are very similar to the estimated effects 
among the full study sample. For example, among the full study sample, the estimated impact 
on degree receipt is 18.3 percentage points — similar in magnitude and statistical significance 
to the 19.8 percentage point estimated impact among survey respondents. This similarity is in 
large part because of the high response rate to the survey; most students in the full sample 
responded to the survey. It is therefore logical that estimated impacts among the full sample 
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and among survey respondents would also be similar. Thus, while the survey results appear 
unlikely to generalize to nonrespondents, they are probably close to representative of the full 
study sample. 

The results in Appendix Table B. 1 and B.2 — combined with the high response rate for 
the survey — provide evidence that, with some caution, impact estimates calculated from 
survey responses can be generalized to all study participants. 

Comparison of Program and Control Group Respondent 

Baseline Characteristics 

A slightly higher proportion of program group students responded to the survey (85 
percent) as compared with control group students (81 percent). Appendix Table B.3 compares 
baseline characteristics for respondents in the program and control groups to determine whether 
respondents’ characteristics differed between the two research groups. The table shows that the 
two groups were comparable, with survey respondents in the program and control groups 
similar on all measured baseline characteristics. 

An omnibus F-test was conducted to examine whether survey respondents’ baseline 
characteristics were jointly predictive of student’s experimental status. The results were not 
statistically significant, indicating little evidence that the groups of respondents were systemati- 
cally different at the outset of the study. This test confinns that among the survey respondents, it 
is reasonable to compare the program and control groups and expect an internally valid causal 
estimate of the effect of ASAP on survey respondents. 


Conclusion 

The response rate for the MDRC student survey was 83 percent. The baseline characteristics of 
program and control group students who responded to the survey were not jointly significantly 
different. The characteristics of survey respondents and nonrespondents were also not jointly 
significantly different. While the estimated impact of ASAP was found to be different among 
those who responded to the survey and those who did not, the estimated impact among survey 
respondents is similar to the estimated impact among the full sample. Thus, the survey results 
are likely representative of the full sample, but may not be representative of the experiences of 
the 17 percent of the sample who did not respond. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Appendix Table B.3 

Characteristics of Program and Control Group Survey Respondents 
Three- Year Impacts Report 


Characteristic (%) 

Number of 
Observations 

All Survey 
Respondents 

Program Control 
Group Group 

P-value 

Female 

742 

63.2 

65.6 

60.7 

0.1674 

Earned a high school diploma or 

higher at 





baseline 11 

742 

75.6 

74.4 

76.8 

0.4464 

Number of developmental courses 





needed at baseline 





0.3798 

0 

660 

2.3 

1.5 

3.1 


1 

660 

40.6 

42.7 

38.4 


2 

660 

49.7 

48.8 

50.6 


3 or more 

660 

7.4 

7.0 

7.9 


Subject of developmental need 





0.7348 

English 

645 

17.5 

16.6 

18.5 


Math 

645 

54.7 

56.1 

53.2 


English and math 

645 

27.8 

27.3 

28.2 


Response rate 

894 

83.0 

85.2 

80.8 * 

0.0800 

Sample size 


742 

384 

358 



SOURCE: MDRC calculations using Baseline Information Form (BIF) data and CUNY Institutional 
Research Database (IRDB) test data. 

NOTES: A two-tailed t-test was applied to differences between control group survey respondents and 
program group survey respondents for gender and dipomas/degrees earned. A chi-squared test was 
applied to differences between the groups for developmental courses and subjects needed. Levels for 
statistically significant differences between program and control groups are indicated as: * = 10 percent; 
** = 5 percent; and *** = 1 percent. 

To analyze whether on average program and control group survey respondents differed from each 
other, an omnibus F-test was performed, which yielded a p-value of 0.81 17. This finding suggests that 
on the baseline characteristics shown above, program and control group survey respondents do not differ 
from one another. 

Estimates are adjusted by site and cohort. 

Missing values are not included in individual variable distributions. 

Distributions may not add to 100 percent because of rounding. 

a This number includes high school diplomas, occupational and technical certificates, and unspecified 
other types of degrees. Not included are students who earned no degree, earned a GED and no other 
degrees, or who are missing degree information. 
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2. Creation of Survey Scales 

This section of the appendix describes two scales reported in Chapter 3 from the MDRC student 
survey. This section provides details on the Quality of Advising scale and the Integration and 
Sense of Belonging scale, including questions that composed the scales and data processing 
conducted to calculate the values presented in Chapter 3. 


Quality of Advising 

The quality of advising measures presented in Table 3.3 are derived from five questions 
administered in the MDRC student survey. These questions were adapted from the Academic 
Advising Inventory (AAI). Students were asked to indicate if they strongly agree (1); agree (2); 
disagree (3); or strongly disagree (4) with the following: 

1. You are satisfied in general with the academic advising you have received. 

2. You have received accurate information about courses, programs, and requirements 
through academic advising. 

3. Academic advisers kept you informed about deadlines related to institutional poli- 
cies and procedures, such as drop/add periods, withdrawal deadlines, registration 
periods, etc. 

4. Academic advising has been available when you needed it. 

5. Sufficient time has been available when you met with academic advisers. 

Originally, for all questions, a higher value response indicates lesser quality advising. 
All five questions were recoded to change the direction of responses. 5 After the direction was 
recoded, a higher value response indicated higher quality advising. 

A scale was created as the unweighted average of a student’s responses to all five ques- 
tions. Respondents had the option to skip or refuse any question in the scale; a small number of 
students responded to some but not all of the five questions in the scale. If the student answered 
only one or two questions in the scale, the scale was not calculated. For students who answered 
three or four questions in the scale, the scale was calculated as the average of that student’s 
responses to the questions that were answered. 

The table presents the percent of students rating the advising they received as high or 
low quality. The low category included students whose calculated score is one or more standard 


5 For all five questions: 1 became 4; 2 became 3; 3 became 2; 4 became 1. 
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deviations below the mean, indicating a relatively lower rating. The high category included 
students whose calculated score is one or more standard deviations above the mean, indicating a 
relatively higher rating. 

Finally, a factor analysis tested how well the items included in the scale measure a 
common underlying construct. The factor analysis yielded a standardized Cronbach’s alpha of 

0.895. 

Integration and Sense of Belonging at School 

The sense of belonging measures presented in Table 3.9 are derived from nine questions asked 
in the MDRC student survey. These questions were first devised for use in MDRC’s evaluation 
of a learning communities program at Kingsborough Community College. 6 Students were asked 
to indicate if they strongly agree (1); agree (2); disagree (3); or strongly disagree (4) with the 
following: 

1 . College is an unfriendly place. 

2. I do not feel that I fit in or belong in college. 

3. The instructors and staff understand who I am and where I am coming from. 

4. It is difficult to make good friends with other students. 

5. The other students do not understand who I am and where I am coming from. 

6. College has the feeling of a community, where many people share the same goals 
and interests. 

7. Many people at college know me by name. 

8. I do not feel I am part of college life. 

9. I feel that I matter to the college instructors, staff, and other students. 

Originally, for some questions, a higher value response indicated a greater sense of be- 
longing, and for some questions, a higher value response indicated a lesser sense of belonging. 
Those latter questions were recoded to change the direction of responses. 7 After the direction 
was recoded for those questions, a higher value response indicated a greater sense of belonging 
for all questions. 


6 Scrivener et al. (2008). 

7 For questions 3, 6, 7, and 9: 1 became 4; 2 became 3; 3 became 2; 4 became 1. 
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A scale is created as the unweighted average of a student’s responses to all nine ques- 
tions. Respondents had the option to skip or refuse any question in the scale; a small number of 
students responded to some but not all of the nine questions in the scale. If the student answered 
five or fewer questions in the scale, the scale was not calculated. For students who answered six 
or more questions in the scale, the scale was calculated as the average of that student’s respons- 
es to the questions that were answered. 

The table presents the percent of students reporting a high or low sense of belonging. 
The low category included students whose calculated score is one or more standard deviations 
below the mean, indicating a lesser integration and sense of belonging. The high category 
included students whose calculated score is one or more standard deviations above the mean, 
indicating a greater integration and sense of belonging. 

Finally, a factor analysis tested how well the items included in the scale measure a 
common underlying construct. The factor analysis yielded a standardized Cronbach’s alpha of 
0 . 846 . 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Appendix Table C.l 

Enrollment at CUNY, Nationally, Full Time, and ASAP Participation 
Three-Year Impacts Report 



Program Group 

Control Group 

Estimated Effects 

Outcome (%) 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Mean 

Difference 

Standard 

Error 

P-Value 

Enrolled at anv CUNY college 

Semester 1 

96.4 

18.5 

94.0 

23.9 

2.5 

1.4 

0.0816 

Main session 

96.4 

18.5 

93.7 

24.3 

2.7 

1.4 

0.0601 

Intersession 

58.4 

49.4 

37.4 

48.5 

21.0 

2.8 

0.0000 

Semester 2 

91.1 

28.5 

80.9 

39.4 

10.3 

2.3 

0.0000 

Main session 

90.3 

29.7 

80.7 

39.5 

9.6 

2.3 

0.0000 

Intersession 

54.0 

49.9 

28.8 

45.4 

25.2 

2.9 

0.0000 

Semester 3 

76.2 

42.6 

68.8 

46.4 

7.4 

3.0 

0.0127 

Main session 

75.8 

42.9 

68.1 

46.7 

7.7 

3.0 

0.0106 

Intersession 

45.0 

49.8 

28.1 

45.0 

16.9 

2.9 

0.0000 

Semester 4 

69.0 

46.3 

59.5 

49.1 

9.5 

3.2 

0.0031 

Main session 

67.7 

46.8 

58.2 

49.4 

9.5 

3.2 

0.0033 

Intersession 

32.9 

47.0 

23.0 

42.2 

9.9 

2.8 

0.0005 

Semester 5 

57.4 

49.5 

50.3 

50.1 

7.1 

3.3 

0.0336 

Main session 

56.5 

49.6 

49.9 

50.1 

6.6 

3.3 

0.0469 

Intersession 

20.0 

40.0 

19.1 

39.4 

0.9 

2.6 

0.7379 

Semester 6 

45.9 

49.9 

41.1 

49.3 

4.8 

3.3 

0.1469 

Main session 

45.0 

49.8 

40.4 

49.1 

4.6 

3.3 

0.1658 

Intersession 

14.5 

35.2 

10.9 

31.3 

3.5 

2.2 

0.1079 

Enrolled at anv college 

Semester 1 

96.7 

18.0 

94.2 

23.5 

2.5 

1.4 

0.0755 

Semester 2 

91.8 

27.5 

81.6 

38.8 

10.3 

2.2 

0.0000 

Semester 3 

77.3 

41.9 

70.2 

45.8 

7.2 

2.9 

0.0144 

Semester 4 

71.2 

45.3 

62.9 

48.4 

8.3 

3.1 

0.0080 

Semester 5 

61.9 

48.6 

55.3 

49.8 

6.6 

3.3 

0.0455 

Semester 6 

51.2 

50.0 

47.4 

50.0 

3.9 

3.3 

0.2491 

Enrolled full time at anv CUNY college 

Semester 1 95.8 20.1 

85.2 

35.6 

10.6 

1.9 

0.0000 

Semester 2 

85.6 

35.2 

65.2 

47.7 

20.4 

2.8 

0.0000 

Semester 3 

73.8 

44.0 

59.6 

49.1 

14.2 

3.1 

0.0000 

Semester 4 

63.0 

48.3 

47.9 

50.0 

15.1 

3.3 

0.0000 

Semester 5 

45.7 

49.9 

39.6 

49.0 

6.1 

3.3 

0.0641 

Semester 6 

32.8 

47.0 

27.4 

44.7 

5.4 

3.0 

0.0773 

Sample size (total = 896) 

451 


445 






(continued) 
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Appendix Table C.l (continued) 


SOURCE: MDRC calculations from CUNY Institutional Research Database (1RDB) and National 
Student Clearinghouse data. 

NOTES: A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site and cohort. 

Enrollment is based on courses in which students are still enrolled as of the end of the add/drop 
period. 

Full-time enrollment is defined as enrollment in 12 or more credits. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Appendix Table C.2 

Credits Attempted: Three Years 

Three-Year Impacts Report 



Program Group 

Control Group 

Estimated Effects 

Outcome 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Mean 

Difference 

Standard 

Error 

P-Value 

Total credits attempted 

Semester 1 

16.10 

4.71 

13.92 

5.04 

2.18 

0.32 

0.0000 

Main session 

12.98 

3.10 

12.09 

3.87 

0.89 

0.23 

0.0001 

Intersession 

3.12 

3.08 

1.83 

2.69 

1.29 

0.17 

0.0000 

Semester 2 

14.50 

5.84 

11.23 

6.50 

3.28 

0.41 

0.0000 

Main session 

11.79 

4.37 

9.86 

5.48 

1.93 

0.33 

0.0000 

Intersession 

2.71 

2.79 

1.37 

2.40 

1.34 

0.16 

0.0000 

Semester 3 

12.29 

7.45 

9.73 

7.37 

2.56 

0.49 

0.0000 

Main session 

10.08 

5.98 

8.39 

6.23 

1.69 

0.40 

0.0000 

Intersession 

2.20 

2.68 

1.34 

2.36 

0.86 

0.15 

0.0000 

Semester 4 

10.17 

7.61 

8.30 

7.58 

1.86 

0.51 

0.0002 

Main session 

8.64 

6.36 

7.25 

6.55 

1.39 

0.43 

0.0012 

Intersession 

1.53 

2.41 

1.06 

2.15 

0.47 

0.14 

0.0010 

Semester 5 

7.48 

7.09 

6.71 

7.24 

0.78 

0.48 

0.1048 

Main session 

6.66 

6.33 

5.89 

6.33 

0.77 

0.42 

0.0669 

Intersession 

0.82 

1.82 

0.81 

1.86 

0.01 

0.12 

0.9556 

Semester 6 

5.73 

6.97 

4.90 

6.49 

0.82 

0.45 

0.0672 

Main session 

5.02 

6.07 

4.38 

5.82 

0.63 

0.40 

0.1102 

Intersession 

0.71 

1.92 

0.52 

1.61 

0.19 

0.12 

0.0984 

Developmental credits attempted 

Semester 1 5.60 

3.83 

3.62 

3.36 

1.98 

0.24 

0.0000 

Main session 

4.00 

3.14 

3.16 

3.12 

0.84 

0.20 

0.0000 

Intersession 

1.60 

2.41 

0.45 

1.44 

1.14 

0.13 

0.0000 

Semester 2 

2.62 

3.29 

1.71 

2.58 

0.90 

0.20 

0.0000 

Main session 

1.82 

2.63 

1.55 

2.45 

0.27 

0.17 

0.1064 

Intersession 

0.80 

1.67 

0.16 

0.87 

0.64 

0.09 

0.0000 

Semester 3 

1.30 

2.55 

0.95 

2.01 

0.35 

0.15 

0.0227 

Main session 

0.98 

2.10 

0.79 

1.77 

0.19 

0.13 

0.1381 

Intersession 

0.32 

1.12 

0.17 

0.92 

0.16 

0.07 

0.0209 

Semester 4 

0.60 

1.74 

0.87 

1.99 

-0.27 

0.12 

0.0316 

Main session 

0.48 

1.47 

0.69 

1.68 

-0.21 

0.11 

0.0510 

Intersession 

0.12 

0.69 

0.18 

0.88 

-0.06 

0.05 

0.2363 

Semester 5 

0.29 

1.24 

0.63 

1.76 

-0.34 

0.10 

0.0008 

Main session 

0.25 

1.05 

0.58 

1.65 

-0.33 

0.09 

0.0004 

Intersession 

0.04 

0.42 

0.05 

0.43 

-0.01 

0.03 

0.7269 


(continued) 
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Appendix Table C.2 (continued) 



Program Group 

Control Group 

Estimated Effects 

Outcome 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Mean 

Difference 

Standard 

Error 

P-Value 

Semester 6 

0.14 

0.79 

0.35 

1.37 

-0.22 

0.07 

0.0036 

Main session 

0.11 

0.69 

0.30 

1.18 

-0.18 

0.06 

0.0043 

Intersession 

0.02 

0.28 

0.06 

0.54 

-0.03 

0.03 

0.2358 

College-level credits attempted 

Semester 1 

10.50 

4.56 

10.30 

4.92 

0.20 

0.30 

0.4960 

Main session 

8.98 

3.58 

8.93 

3.90 

0.06 

0.24 

0.8111 

Intersession 

1.52 

2.17 

1.37 

2.28 

0.14 

0.14 

0.2939 

Semester 2 

11.88 

5.58 

9.51 

6.03 

2.37 

0.38 

0.0000 

Main session 

9.97 

4.35 

8.31 

5.01 

1.67 

0.31 

0.0000 

Intersession 

1.91 

2.50 

1.21 

2.26 

0.71 

0.15 

0.0000 

Semester 3 

10.99 

7.10 

8.78 

6.85 

2.21 

0.46 

0.0000 

Main session 

9.11 

5.67 

7.61 

5.85 

1.50 

0.38 

0.0001 

Intersession 

1.88 

2.52 

1.17 

2.16 

0.71 

0.14 

0.0000 

Semester 4 

9.57 

7.35 

7.44 

6.95 

2.13 

0.48 

0.0000 

Main session 

8.16 

6.19 

6.56 

6.09 

1.59 

0.41 

0.0001 

Intersession 

1.41 

2.32 

0.88 

1.96 

0.54 

0.14 

0.0001 

Semester 5 

7.20 

6.85 

6.08 

6.77 

1.12 

0.45 

0.0140 

Main session 

6.42 

6.17 

5.31 

5.89 

1.10 

0.40 

0.0060 

Intersession 

0.78 

1.74 

0.76 

1.81 

0.02 

0.12 

0.8864 

Semester 6 

5.59 

6.87 

4.55 

6.17 

1.04 

0.44 

0.0171 

Main session 

4.90 

5.97 

4.09 

5.53 

0.82 

0.38 

0.0335 

Intersession 

0.69 

1.91 

0.46 

1.52 

0.23 

0.11 

0.0465 

Sample size (total = 896) 

451 


445 






SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site and cohort. 

Measures of credits earned in main sessions or intersessions do not exclude courses passed more than 
once. However, measures of credits earned in semesters exclude courses that are passed more than once. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Appendix Table C.3 

Credits Earned: Three Years 

Three- Year Impacts Report 



Program Group 

Control Group 

Estimated Effects 

Outcome 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Mean 

Difference 

Standard 

Error 

P- Value 

Total credits earned 

Semester 1 

11.36 

5.78 

9.31 

6.38 

2.05 

0.40 

0.0000 

Main session 

9.42 

4.61 

7.95 

5.21 

1.47 

0.33 

0.0000 

Intersession 

1.97 

2.64 

1.37 

2.31 

0.61 

0.15 

0.0001 

Semester 2 

10.10 

6.77 

7.86 

6.69 

2.24 

0.45 

0.0000 

Main session 

8.14 

5.29 

6.73 

5.62 

1.41 

0.36 

0.0001 

Intersession 

1.97 

2.51 

1.13 

2.16 

0.84 

0.15 

0.0000 

Semester 3 

8.92 

7.07 

7.19 

6.77 

1.73 

0.46 

0.0002 

Main session 

7.33 

5.82 

6.08 

5.73 

1.26 

0.38 

0.0011 

Intersession 

1.58 

2.36 

1.11 

2.12 

0.47 

0.14 

0.0009 

Semester 4 

7.62 

7.02 

6.05 

6.72 

1.56 

0.46 

0.0007 

Main session 

6.42 

5.91 

5.24 

5.85 

1.18 

0.39 

0.0026 

Intersession 

1.19 

2.14 

0.81 

1.91 

0.38 

0.13 

0.0034 

Semester 5 

5.47 

6.13 

4.92 

6.13 

0.55 

0.41 

0.1766 

Main session 

4.85 

5.52 

4.31 

5.43 

0.54 

0.36 

0.1367 

Intersession 

0.62 

1.53 

0.61 

1.59 

0.01 

0.10 

0.9121 

Semester 6 

4.33 

6.06 

3.83 

5.61 

0.50 

0.39 

0.2027 

Main session 

3.79 

5.30 

3.38 

5.01 

0.41 

0.34 

0.2345 

Intersession 

0.55 

1.66 

0.46 

1.50 

0.09 

0.10 

0.3871 

Developmental credits earned 

Semester 1 2.88 

3.14 

1.75 

2.69 

1.13 

0.19 

0.0000 

Main session 

2.21 

2.84 

1.52 

2.47 

0.69 

0.17 

0.0001 

Intersession 

0.69 

1.71 

0.23 

1.00 

0.47 

0.09 

0.0000 

Semester 2 

1.11 

2.15 

0.80 

1.87 

0.31 

0.13 

0.0226 

Main session 

0.71 

1.75 

0.69 

1.70 

0.02 

0.11 

0.8361 

Intersession 

0.40 

1.26 

0.11 

0.68 

0.29 

0.07 

0.0000 

Semester 3 

0.51 

1.62 

0.43 

1.50 

0.08 

0.10 

0.4247 

Main session 

0.37 

1.37 

0.34 

1.25 

0.02 

0.09 

0.7803 

Intersession 

0.15 

0.80 

0.09 

0.72 

0.06 

0.05 

0.2447 

Semester 4 

0.21 

0.98 

0.47 

1.54 

-0.26 

0.09 

0.0024 

Main session 

0.18 

0.91 

0.38 

1.34 

-0.20 

0.08 

0.0089 

Intersession 

0.03 

0.37 

0.09 

0.64 

-0.06 

0.04 

0.0773 

Semester 5 

0.09 

0.64 

0.23 

1.05 

-0.14 

0.06 

0.0184 

Main session 

0.08 

0.61 

0.21 

1.02 

-0.12 

0.06 

0.0280 

Intersession 

0.01 

0.19 

0.02 

0.28 

-0.01 

0.02 

0.3850 
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Appendix Table C.3 (continued) 



Program Group 

Control Group 

Estimated Effects 

Outcome 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Mean 

Difference 

Standard 

Error 

P- Value 

Semester 6 

0.00 

0.00 

0.15 

0.88 

-0.15 

0.04 

0.0003 

Main session 

0.00 

0.00 

0.12 

0.73 

-0.12 

0.03 

0.0007 

Intersession 

0.00 

0.00 

0.03 

0.43 

-0.03 

0.02 

0.0869 

College-level credits earned 

Semester 1 

8.49 

4.95 

7.56 

5.58 

0.92 

0.34 

0.0065 

Main session 

7.21 

4.07 

6.42 

4.54 

0.78 

0.28 

0.0055 

Intersession 

1.28 

2.01 

1.14 

2.03 

0.14 

0.13 

0.2723 

Semester 2 

8.99 

6.27 

7.05 

6.16 

1.93 

0.41 

0.0000 

Main session 

7.43 

5.01 

6.04 

5.19 

1.39 

0.34 

0.0000 

Intersession 

1.56 

2.30 

1.02 

2.06 

0.55 

0.14 

0.0001 

Semester 3 

8.41 

6.87 

6.76 

6.46 

1.65 

0.44 

0.0002 

Main session 

6.97 

5.62 

5.73 

5.49 

1.23 

0.37 

0.0009 

Intersession 

1.44 

2.25 

1.02 

1.99 

0.41 

0.13 

0.0019 

Semester 4 

7.41 

6.90 

5.58 

6.30 

1.83 

0.44 

0.0000 

Main session 

6.24 

5.82 

4.86 

5.52 

1.38 

0.38 

0.0003 

Intersession 

1.16 

2.11 

0.72 

1.80 

0.44 

0.13 

0.0004 

Semester 5 

5.38 

6.05 

4.69 

5.90 

0.69 

0.40 

0.0837 

Main session 

4.77 

5.47 

4.11 

5.21 

0.66 

0.35 

0.0615 

Intersession 

0.61 

1.50 

0.58 

1.57 

0.03 

0.10 

0.8027 

Semester 6 

4.33 

6.06 

3.68 

5.43 

0.65 

0.38 

0.0919 

Main session 

3.79 

5.30 

3.26 

4.88 

0.53 

0.34 

0.1221 

Intersession 

0.55 

1.66 

0.42 

1.45 

0.13 

0.10 

0.2239 

Sample size (total = 896) 

451 


445 






SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site and cohort. 

Measures of credits earned in main sessions or intersessions do not exclude courses passed more 
than once. However, measures of credits earned in semesters exclude courses that are passed more 
than once. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Appendix Table C.4 

Credits Attempted: Three Years (Cumulative) 

Three- Year Impacts Report 



Program Group 

Control Group 

Estimated Effects 

Outcome 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Mean 

Difference 

Standard 

Error 

P- Value 

Sessions enrolled 

Semester 1 (out of 2) 

1.55 

0.57 

1.31 

0.58 

0.24 

0.03 

0.0000 

Semester 2 (out of 4) 

2.99 

1.03 

2.41 

1.10 

0.58 

0.07 

0.0000 

Semester 3 (out of 6) 

4.20 

1.59 

3.37 

1.64 

0.83 

0.10 

0.0000 

Semester 4 (out of 8) 

5.21 

2.13 

4.18 

2.18 

1.02 

0.14 

0.0000 

Semester 5 (out of 10) 

5.97 

2.59 

4.87 

2.69 

1.10 

0.17 

0.0000 

Semester 6 (out of 12) 

6.57 

2.99 

5.38 

3.07 

1.18 

0.20 

0.0000 

Total credits attempted 

Semester 1 

16.10 

4.71 

13.92 

5.04 

2.18 

0.32 

0.0000 

Semester 2 

30.60 

8.93 

25.14 

9.93 

5.46 

0.62 

0.0000 

Semester 3 

42.89 

14.34 

34.87 

15.21 

8.01 

0.97 

0.0000 

Semester 4 

53.06 

19.71 

43.18 

20.81 

9.88 

1.34 

0.0000 

Semester 5 

60.54 

24.07 

49.89 

25.58 

10.65 

1.65 

0.0000 

Semester 6 

66.27 

28.01 

54.79 

29.29 

11.48 

1.90 

0.0000 

Developmental credits attempted 

Semester 1 5.60 

3.83 

3.62 

3.36 

1.98 

0.24 

0.0000 

Semester 2 

8.22 

5.73 

5.33 

4.43 

2.89 

0.34 

0.0000 

Semester 3 

9.52 

7.02 

6.28 

4.97 

3.23 

0.40 

0.0000 

Semester 4 

10.11 

7.72 

7.15 

5.55 

2.97 

0.45 

0.0000 

Semester 5 

10.40 

8.07 

7.78 

6.13 

2.63 

0.48 

0.0000 

Semester 6 

10.54 

8.24 

8.13 

6.59 

2.41 

0.50 

0.0000 

College-level credits attempted 

Semester 1 10.50 

4.56 

10.30 

4.92 

0.20 

0.30 

0.4960 

Semester 2 

22.39 

8.72 

19.81 

9.49 

2.57 

0.59 

0.0000 

Semester 3 

33.37 

13.81 

28.59 

14.39 

4.78 

0.92 

0.0000 

Semester 4 

42.94 

19.00 

36.03 

19.56 

6.91 

1.27 

0.0000 

Semester 5 

50.14 

23.29 

42.11 

23.85 

8.03 

1.56 

0.0000 

Semester 6 

55.73 

27.20 

46.66 

27.41 

9.07 

1.81 

0.0000 

Sample size (total = 896) 

451 


445 
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Appendix Table C.4 (continued) 


SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site and cohort. 

Measures of cumulative credits earned exclude courses that are passed more than once. 
Enrollment is based on courses in which students are still enrolled as of the end of the add/drop 
period. 
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Evaluation of Accelerated Study in Associate Programs (ASAP) for 
Developmental Education Students 

Appendix Table C.5 

Credits Earned: Three Years (Cumulative) 

Three- Year Impacts Report 



Program Group 

Control Group 

Estimated Effects 

Outcome 

Mean 

Standard 

Deviation 

Mean 

Standard 

Deviation 

Mean 

Difference 

Standard 

Error 

P- Value 

Sessions enrolled 

Semester 1 (out of 2) 

1.55 

0.57 

1.31 

0.58 

0.24 

0.03 

0.0000 

Semester 2 (out of 4) 

2.99 

1.03 

2.41 

1.10 

0.58 

0.07 

0.0000 

Semester 3 (out of 6) 

4.20 

1.59 

3.37 

1.64 

0.83 

0.10 

0.0000 

Semester 4 (out of 8) 

5.21 

2.13 

4.18 

2.18 

1.02 

0.14 

0.0000 

Semester 5 (out of 10) 

5.97 

2.59 

4.87 

2.69 

1.10 

0.17 

0.0000 

Semester 6 (out of 12) 

6.57 

2.99 

5.38 

3.07 

1.18 

0.20 

0.0000 

Total credits earned 

Semester 1 

11.36 

5.78 

9.31 

6.38 

2.05 

0.40 

0.0000 

Semester 2 

21.45 

10.99 

17.13 

11.87 

4.31 

0.75 

0.0000 

Semester 3 

30.34 

16.52 

24.31 

17.24 

6.03 

1.12 

0.0000 

Semester 4 

37.93 

21.57 

30.35 

22.27 

7.58 

1.45 

0.0000 

Semester 5 

43.37 

25.36 

35.22 

26.42 

8.15 

1.72 

0.0000 

Semester 6 

47.68 

28.87 

39.01 

29.86 

8.67 

1.95 

0.0000 

Developmental credits earned 

Semester 1 2.88 

3.14 

1.75 

2.69 

1.13 

0.19 

0.0000 

Semester 2 

3.98 

3.72 

2.54 

3.24 

1.43 

0.23 

0.0000 

Semester 3 

4.49 

3.95 

2.97 

3.47 

1.52 

0.25 

0.0000 

Semester 4 

4.69 

4.00 

3.44 

3.77 

1.25 

0.26 

0.0000 

Semester 5 

4.79 

4.03 

3.67 

4.02 

1.11 

0.27 

0.0000 

Semester 6 

4.79 

4.03 

3.82 

4.13 

0.96 

0.27 

0.0004 

College-level credits earned 

Semester 1 

8.49 

4.95 

7.56 

5.58 

0.92 

0.34 

0.0065 

Semester 2 

17.47 

9.93 

14.59 

10.63 

2.88 

0.67 

0.0000 

Semester 3 

25.85 

15.43 

21.34 

15.75 

4.51 

1.02 

0.0000 

Semester 4 

33.24 

20.41 

26.91 

20.57 

6.33 

1.35 

0.0000 

Semester 5 

38.59 

24.17 

31.55 

24.46 

7.04 

1.61 

0.0000 

Semester 6 

42.89 

27.70 

35.18 

27.80 

7.71 

1.84 

0.0000 

Sample size (total = 896) 

451 


445 
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Appendix Table C.5 (continued) 


SOURCE: MDRC calculations from CUNY Institutional Research Database (IRDB) data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. 

Estimates are adjusted by site and cohort. 

Measures of cumulative credits earned exclude courses that are passed more than once. 
Enrollment is based on courses in which students are still enrolled as of the end of the add/ drop 
period. 
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NOTE: A complete publications list is available from MDRC and on its website (www.mdrc.org), 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization 
dedicated to learning what works to improve the well-being of low-income people. 
Through its research and the active communication of its findings, MDRC seeks to 
enhance the effectiveness of social and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is 
best known for mounting rigorous, large-scale, real-world tests of new and existing 
policies and programs. Its projects are a mix of demonstrations (field tests of 
promising new program approaches) and evaluations of ongoing government and 
community initiatives. MDRC’s staff bring an unusual combination of research and 
organizational experience to their work, providing expertise on the latest in qualitative 
and quantitative methods and on program design, development, implementation, and 
management. MDRC seeks to learn not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s 
findings in the broader context of related research — in order to build knowledge 
about what works across the social and education policy fields. MDRC’s findings, 
lessons, and best practices are proactively shared with a broad audience in the policy 
and practitioner community as well as with the general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range 
of policy areas and target populations. Once known primarily for evaluations 
of state welfare-to-work programs, today MDRC is also studying public school 
reforms, employment programs for ex-offenders and people with disabilities, and 
programs to help low-income students succeed in college. MDRC’s projects are 
organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the 
United Kingdom, MDRC conducts its projects in partnership with national, state, and 
local governments, public school systems, community organizations, and numerous 
private philanthropies. 


